Image output device

ABSTRACT

The present invention relates to an image output device mounted on a vehicle to implement augmented reality. One or more of an autonomous driving vehicle, a user terminal, and a server of the present invention can be linked to an artificial intelligence module, a drone (unmanned aerial vehicle, UAV), a robot, an augmented reality (AR) device, a virtual reality (VR) device, a device related to 5G services, etc.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a National Stage application under 35 U.S.C. § 371of International Application No. PCT/KR2020/001406, filed on Jan. 30,2020, which claims the benefit of Korean Application No.10-2019-0062696, filed on May 28, 2019, and U.S. Provisional ApplicationNo. 62/799,693, filed on Jan. 31, 2019. The disclosures of the priorapplications are incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to an image output device provided in avehicle to enable augmented reality, and a method for controlling thesame.

BACKGROUND ART

A vehicle refers to means of transporting people or goods by usingkinetic energy. Representative examples of vehicles include automobilesand motorcycles.

For safety and convenience of a user who uses the vehicle, varioussensors and devices are provided in the vehicle, and functions of thevehicle are diversified.

The functions of the vehicle may be divided into a convenience functionfor promoting driver's convenience, and a safety function for enhancingsafety of the driver and/or pedestrians.

First, the convenience function has a development motive associated withthe driver's convenience, such as providing infotainment(information+entertainment) to the vehicle, supporting a partiallyautonomous driving function, or helping the driver ensuring a field ofvision at night or at a blind spot. For example, the conveniencefunctions may include various functions, such as an active cruisecontrol (ACC), a smart parking assist system (SPAS), a night vision(NV), a head up display (HUD), an around view monitor (AVM), an adaptiveheadlight system (AHS), and the like.

The safety function is a technique of ensuring safeties of the driverand/or pedestrians, and may include various functions, such as a lanedeparture warning system (LDWS), a lane keeping assist system (LKAS), anautonomous emergency braking (AEB), and the like.

In order to further improve the convenience functions and the safetyfunctions, a vehicle-specific communication technology is beingdeveloped. For example, a vehicle to infrastructure (V2I) that enablescommunication between a vehicle and an infrastructure, a Vehicle toVehicle (V2V) that enables communication between vehicles, a Vehicle toEverything (V2X) that enables communication between a vehicle and anobject, and the like.

An image output device for visually providing occupants or passengers onboard with various information may be disposed at a vehicle. The imageoutput device includes a head-up display (HUD) that presents informationthrough a windshield of a vehicle or a separately provided transparentscreen, and/or various displays that output information through a panel.

The image output device is evolving into a way to provide routenavigation information to the destination and information regarding apoint of interest (POI) while effectively providing various information.In particular, a research has been conducted to provide an image outputdevice that can directly and effectively provide necessary informationin a manner that does not interfere with driving of the driver who needsto pay attention while driving.

SUMMARY

The present disclosure is directed to solving the aforementionedproblems and other drawbacks.

The present disclosure describes an image output device that can providean occupant on board a vehicle with various information using augmentedreality, and a method for controlling the same.

The present disclosure also describes an image output device that canenable an occupant on board a vehicle to display driving informationcollected from other vehicles using augmented reality.

According to one aspect of the subject matter described in thisapplication, an image output device is provided in a vehicle to enableaugmented reality. The device includes the image output device includesa controller configured to receive, in real time, a forward imagecapturing an image in front of the vehicle, search for one or more laneson which the vehicle is expected to travel in the forward image,generate image information including a carpet image or images indicatingthe searched one or more lanes in lane units, and transmit the imageinformation to an image output unit outputting visual information sothat the image information is output to the image output device. Thecontroller is configured to receive, in real time, an image capturedfrom another vehicle present on a route on which the vehicle is expectedto travel, and to generate image information in which the carpet imageis superimposed on at least one of the forward image and the imagecaptured from the another vehicle.

Implementations according to this aspect may include one or more of thefollowing features. For example, the controller may generate imageinformation in which the carpet image is superimposed on the forwardimage and the image captured from the another vehicle.

In some implementations, when the forward image and the image capturedfrom the another vehicle are at least partially the same, the controllermay combine the forward image and the image captured from the anothervehicle, and generate image information in which the carpet image issuperimposed on the combined image.

In some implementations, the controller may receive a route on which theanother vehicle is expected to travel from the another vehicle, andgenerate image information in which a first carpet image indicating theroute on which the vehicle is expected to travel and a second carpetimage indicating the route on which the another vehicle is expected totravel are superimposed on the combined image.

In some implementations, the controller may generate image informationin which a third carpet image having a different shape from the firstand second carpet images is superimposed on the combined image when thevehicle and the another vehicle are expected to travel on the sameroute.

In some implementations, when the route of at least one of the vehicleand the another vehicle is changed while the third carpet image isdisplayed on the image output unit, the controller may stop thegeneration of the image information including the third carpet image,and create an image in which the first and second carpet images aresuperimposed on the combined image.

In some implementations, the controller may receive a route on which theanother vehicle is expected to travel from the another vehicle, andgenerate image information in which a first carpet image indicating theroute on which the vehicle is expected to travel is superimposed on theforward image and a second carpet image indicating the route on whichthe another vehicle is expected to travel is superimposed on the imagecaptured from the another vehicle.

In some implementations, the controller may generate image informationin which the first carpet image and the second carpet image aresuperimposed on the image captured from the another vehicle.

In some implementations, the controller may generate image informationin which the first carpet image and the second carpet image aresuperimposed on the forward image.

In some implementations, the controller may receive informationregarding the another vehicle from the another vehicle, generate imageinformation including the information regarding the another vehicle andthe image captured from the another vehicle, and transmit the generatedimage information to the image output unit

In some implementations, the controller may receive a request for imagesharing of the another vehicle from a driver of the vehicle, generate amessage for indicating the use of points upon receiving the imagesharing request, and transmit the generated message to the image outputunit

In some implementations, the controller may transmit information aboutpoints paid to the another vehicle, so as to receive the image capturedfrom the another vehicle.

In some implementations, the controller may receive, in real time,images captured from a plurality of vehicles present on the route onwhich the vehicle is expected to travel, transmit the forward image andthe images captured from the plurality of vehicles to the image outputunit, and generate image information in which the carpet image issuperimposed on at least one of the forward image and each of the imagescaptured from the plurality of vehicles.

In some implementations, the controller may generate image informationsuch that the images captured from the plurality of vehicles aredisplayed in a smaller size than the forward image.

According to another aspect, an image output device is provided in avehicle to enable augmented reality. The device includes: an imageoutput unit configured to output visual information for enabling theaugmented reality; a communication unit configured to communicate withanother vehicle and a server, and receive, in real time, a forward imagecapturing an image in front of the vehicle; and a controller configuredto control the image output unit to search for one or more lanes onwhich the vehicle is expected to travel in the forward image, and outputa carpet image or images indicating the searched one or more lanes inlane units. The controller is configured to control the communicationunit such that an image captured from another vehicle present on a routeon which the vehicle is expected to travel is received in real time, andcontrol the image output unit such that image information in which thecarpet image is superimposed on at least one of the forward image andthe image captured from the another vehicle is generated.

According to another aspect, a method for controlling an image outputdevice provided in a vehicle to enable augmented reality includes:receiving, in real time, a forward image capturing an image in front ofthe vehicle; searching for one or more lanes on which the vehicle isexpected to travel in the forward image; receiving, in real time, animage captured from another vehicle present on a route on which thevehicle is expected to travel; generating image information including acarpet image or images indicating the searched one or more lanes in laneunits using at least one of the forward image and the image capturedfrom the another vehicle; and transmitting the image information to animage output unit. The image information includes an image in which thecarpet image is superimposed on at least one of the forward image andthe image captured from the another vehicle.

Implementations according to this aspect may include one or more of thefollowing features. For example, the image information may include animage in which the carpet image is superimposed on the forward image andthe image captured from the another vehicle.

In some implementations, the present disclosure may further providecombining the forward image and the image captured from the anothervehicle when the forward image and the image captured from the anothervehicle are at least partially the same and transmitting the combinedimage to the image output unit.

In some implementations, the present disclosure may further providereceiving a route on which the another vehicle is expected to travelfrom the another vehicle, generating image information in which a firstcarpet image indicating the route on which the vehicle is expected totravel and a second carpet image indicating the route on which theanother vehicle is expected to travel are superimposed on the combinedimage, and transmitting the generated image information to the imageoutput unit.

In some implementations, the present disclosure may further providegenerating image information in which a third carpet image having adifferent shape from the first and second carpet images is superimposedon the combined image and transmitting the generated image informationto the image output unit when the vehicle and the another vehicle areexpected to travel on the same route.

An image output device and a method for controlling the same accordingto implementations of the present disclosure may provide the followingbenefits.

According to the implementations of the present discourse, an occupanton board a vehicle may be provided with a path or route information ofthe vehicle driven by automatous driving or by a driver in lane units(or lane-by-lane) through a carpet image.

In addition, an occupant on board a vehicle may be provided with morevarious driving information through image information collected fromother vehicles traveling ahead of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an AI device according to one implementation of thepresent disclosure;

FIG. 2 illustrates an AI server according to one implementation of thepresent disclosure;

FIG. 3 illustrates an AI system according to one implementation of thepresent disclosure;

FIG. 4 illustrates an outer appearance of a vehicle according to animplementation of the present disclosure;

FIG. 5 illustrates a vehicle exterior from various angles according toan implementation of the present disclosure;

FIGS. 6 and 7 illustrate a vehicle interior according to animplementation of the present disclosure;

FIGS. 8 and 9 are views referenced to describe objects according to animplementation of the present disclosure;

FIG. 10 is a block diagram illustrating a vehicle according to animplementation of the present disclosure;

FIG. 11 is a conceptual diagram illustrating an image output deviceaccording to one implementation of the present disclosure;

FIG. 12 is a schematic view illustrating a communication method forsharing images between vehicles;

FIG. 13 is a schematic view illustrating image sharing between vehicles;

FIG. 14 is a flowchart of an exemplary method of image sharing betweenvehicles;

FIG. 15 is a flowchart of an exemplary method of using points for imagesharing between vehicles;

FIG. 16 is a conceptual diagram illustrating data transmission andreception between vehicles;

FIG. 17 is a flowchart of an exemplary method of calibrating a vehiclecamera;

FIGS. 18 and 19 are schematic views illustrating an example ofdisplaying images received from a plurality of vehicles together;

FIG. 20 is a flowchart of synthesizing a plurality of images receivedfrom other vehicles; and

FIG. 21 is a schematic view illustrating an example in which a specificimage is displayed in a larger size according to user selection.

DETAILED DESCRIPTION

Description will now be given in detail according to one or moreimplementations disclosed herein, with reference to the accompanyingdrawings. In the drawings, the same or similar elements are designatedwith the same or similar reference numerals, and redundant descriptionhas been omitted. The suffixes “module” and “unit” for components orelements used in the following description are given or mixed inconsideration of ease in creating specification, and do not havedistinct meanings or roles. In describing implementations, if a detailedexplanation for a related known technology or construction is consideredto unnecessarily divert the main point, such explanation has beenomitted but would be understood by those skilled in the art. Also, itshould be understood that the accompanying drawings are merelyillustrated to easily explain the concept, and therefore, they shouldnot be construed to limit the technological concept disclosed herein bythe accompanying drawings, and the concept should be construed as beingextended to all modifications, equivalents, and substitutes included inthe concept and technological scope.

Terms including ordinal numbers such as first and second may be used todescribe various elements, but the elements are not limited by theterms. The terms are used merely for the purpose to distinguish anelement from another element.

It will be understood that when an element is referred to as being“connected with” another element, the element can be directly connectedwith the other element or intervening elements may also be present. Onthe contrary, in case where an element is “directly connected” or“directly linked” to another element, it should be understood that anyother element is not existed therebetween.

Singular expressions include plural expressions unless the contextclearly indicates otherwise.

Terms “include” or “has” used herein should be understood that they areintended to indicate the existence of a feature, a number, a step, aconstituent element, a component or a combination thereof disclosed inthe specification, and it may also be understood that the existence oradditional possibility of one or more other features, numbers, steps,elements, components or combinations thereof are not excluded inadvance.

A vehicle disclosed herein may include various types of automobiles suchas cars, motorcycles, and the like. Hereinafter, the vehicle will bedescribed based on a car.

Artificial intelligence (AI) is the field of study devoted to makingmachines intelligent or a methodology to create it, and machine learningis the study of defining and solving various problems dealt with in thefield of the artificial intelligence. Machine learning is also definedas an algorithm that improve the performance of a task throughexperience.

An artificial neural network (ANN) is a (computational) model used inmachine learning, which may refer in general to a model withproblem-solving capabilities that consists of artificial neurons (nodes)forming a network by synaptic connections. The ANN may be defined by aconnection pattern between neurons of different layers, a learningprocess of updating model parameters, and an activation function forgenerating an output value.

The ANN may include an input layer, an output layer, and optionally oneor more hidden layers. Each layer includes one or more neurons, and theANN may include synapses for connecting neurons. In the ANN, each neuroncan output a function value of the activation function for inputsignals, weights, and biases input through the synapse.

Model parameters refer to parameters determined through learning, andinclude a weight value of synaptic connection and biases of neurons. Ahyperparameter refers to a parameter that should be set prior tolearning in a machine learning algorithm, which includes a learningrate, the number of repeats, a mini-batch size, an initializationfunction, and the like.

The purpose of training the ANN may be for determining a model parameterthat can minimize a loss function. The loss function may be used as anindex for determining an optimal model parameter in the training processof the ANN.

Machine learning can be classified into supervised learning,unsupervised learning, and reinforcement learning according to alearning method.

The supervised learning may refer to a method of training an ANN usinglabeled training data, and a label may indicate the correct answer (orresult value) that the ANN must infer when training data is inputthereto. The unsupervised learning may refer to a method of training anANN using unlabeled data. The reinforcement learning may refer to amethod for training an agent defined in a certain environment to selectactions or a sequence of selecting actions that maximizes cumulativerewards in each state.

Machine learning implemented as a deep neural network (DNN) including aplurality of hidden layers is referred to as deep learning, which is aclass of machine learning. Machine learning used herein includes deeplearning.

A robot is a machine designed to automatically handles one or more tasksby its own ability or be automatically operated. In particular, a robotcapable of carrying out a series of actions by recognizing theenvironment and making a decision by itself may be referred to as anintelligent (or smart) robot.

Robots can be classified into an industrial robot, a medical robot, ahousehold robot, a military robot, and the like depending on the purposeof use or field.

The robot may be equipped with a drive (or driving) unit that includesan actuator or a motor to allow the robot to perform various physicaloperations or actions such as moving joints. In addition, a mobile ormoving robot includes a wheel, a brake, a propeller, and the like in adrive unit to thereby travel on the ground or fly in the air.

Autonomous driving refers to a self-driving technology, and anautonomous vehicle (or self-driving vehicle) refers to a vehicle that isdriven without a user's manipulation or with a user's minimalmanipulation.

For example, the autonomous driving may include a technology formaintaining a driving lane, a technology for automatically adjustingspeed such as adaptive cruise control, a technology for automaticallydriving or traveling along a predetermined route or path, and atechnology for automatically setting a path to travel when a destinationis set.

A vehicle includes all of a vehicle including only an internalcombustion engine, a hybrid vehicle including an internal combustionengine and an electric motor, and an electric vehicle including only anelectric motor, and may include not only automobiles, but also trainsand motorcycles.

Here, the autonomous vehicle may be a robot having an autonomous drivingfunction.

Extended reality collectively refers to virtual reality (VR), augmentedreality (AR), and mixed reality (MR). VR technology provides real-worldobjects backgrounds, or the like in a CG image, AR technology providesvirtually created CG images on real-world object images, and MRtechnology is a computer graphic technology that mixes and combinesvirtual objects into the real world.

The MR technology is similar to the AR technology in the sense thatreal-world objects and virtual objects are shown together. However, inthe AR technology, a virtual object is used in the form of complementinga real-world object. On the other hand, in the MR technology, a virtualobject and a real-world are used in an equal manner.

The XR technology may be applied to a head-mount display (HMD), ahead-up display (HUD), a mobile phone, a tablet PC, a laptop, a desktop,a TV, digital signage, and the like, and a device that uses the XRtechnology may be referred to as an “XR device”.

FIG. 1 illustrates an AI device according to one implementation of thepresent disclosure.

An AI device 1000 may be configured as a fixed (or stationary) device ora movable (or mobile) device such as a TV, a projector, a mobile phone,a smartphone, a desktop computer, a notebook computer, a digitalbroadcasting terminal, a personal digital assistant (PDA), a portablemultimedia player (PMP), a navigation device, a tablet PC, a wearabledevice, and a set-top box (STB), a DMB receiver, a radio, a washingmachine, a refrigerator, a desktop computer, digital signage, a robot, avehicle, and the like.

As illustrated in FIG. 1, the AI device 1000 may include a communicationunit 1100, an input unit 1200, a learning processor 1300, a sensing unit1400, an output unit 1500, a memory 1700, and a processor 1800.

The communication unit 1100 may transmit and receive data to and fromexternal devices, such as other AI devices 100 a to 100 e and an AIserver 200, using wired/wireless communication technologies. Forexample, the communication unit 1100 may transmit and receive sensorinformation, a user input, a learning model, a control signal, and thelike with the external devices.

Here, the communication unit 1100 uses communication technologiesincluding Global System for Mobile communication (GSM), Code DivisionMulti Access (CDMA), Long Term Evolution (LTE), 5G, Wireless LAN (WLAN),and Wireless-Fidelity (Wi-Fi), Bluetooth™, Radio FrequencyIdentification (RFID), Infrared Data Association (IrDA), ZigBee, NearField Communication (NFC), and the like.

The input unit 1200 may acquire various types of data.

Here, the input unit 1200 may include a camera for inputting an image(or video) signal, a microphone for receiving an audio signal, and auser input unit for receiving information from a user. When the cameraor the microphone is considered as a sensor, a signal obtained from thecamera or the microphone may be referred to as sensing data or sensorinformation.

The input unit 1200 may acquire training data for model training andinput data to be used when acquiring an output by using a learningmodel. The input unit 1200 may obtain unprocessed or raw input data.Here, the processor 1800 or the learning processor 1300 may extract aninput feature as pre-processing for the input data.

The learning processor 1300 may train a model that consists of anartificial neural network by using training data. Here, the trainedartificial neural network may be referred to as a “learning model”. Thelearning model may be used to infer a result value for new input data,not training data, and the inferred value may be used as a basis fordetermining to perform a specific operation (or action).

The learning processor 1300 may perform AI processing together with alearning processor 240 of the AI server 200.

The learning processor 1300 may include a memory integrated orimplemented in the AI device 1000. Alternatively, the learning processor1300 may be implemented using the memory 1700, an external memorydirectly coupled to the AI device 1000, or a memory kept in an externaldevice.

The sensing unit 1400 may use various sensors to acquire at least one ofinternal information of the AI device 1000, surrounding environmentinformation of the AI device 1000, and user information.

The sensing unit 1400 may include, for example, a proximity sensor, anillumination sensor, an acceleration sensor, a magnetic sensor, agyroscope (or gyro) sensor, an inertial sensor, an RGB sensor, an IRsensor, a finger scan sensor, an ultrasonic sensor, an optical sensor, amicrophone, a LiDAR, a radar, and the like.

The output unit 1500 may generate an output related to a visual,audible, or tactile signal.

In this case, the output unit 1500 may include a display module or unitfor outputting visual information, a speaker for outputting auditoryinformation, a haptic module for outputting tactile information, and thelike.

The memory 1700 may store data that supports various functions orfeatures of the AI device 1000. For example, the memory 1700 may storeinput data acquired from the input unit 1200, training data, a learningmodel, a learning history, and the like.

The processor 1800 may determine at least one executable operation ofthe AI device 1000 based on information determined or generated using adata analysis algorithm or a machine learning algorithm. In addition,the processor 1800 may control the components of the AI device 1000 toperform the determined operation.

To this end, the processor 1800 may request, search, receive, or utilizedata of the learning processor 1300 or the memory 1700. The processor1800 may control the components of the AI device 1000 to perform apredicted (or expected) or desirable operation among the at least oneexecutable operation.

When connection of an external device is required to perform thedetermined operation, the processor 1800 may generate a control signalfor controlling the external device and transmit the generated controlsignal to the external device.

The processor 1800 may obtain intention or intent informationcorresponding to a user input to determine a user's requirement (orrequest) based on the obtained intent information.

Here, the processor 1800 may obtain intent information corresponding tothe user input by using at least one of a Speech to Text (STT) enginefor converting a voice or audio input into a text string and a naturallanguage processing (NLP) engine for obtaining intent information of anatural language.

At least one of the STT engine and the NLP engine may, at leastpartially, consist of an artificial neural network trained according toa machine learning algorithm. In addition, at least one of the STTengine and the NLP engine may be trained by the learning processor 1300,trained by the learning processor 240 of the AI server 200, or trainedby distributed processing thereof.

The processor 1800 may collect history information including operationcontents of the AI device 1000 or user's feedback on an operation of theAI device 1000, and store the history information in the memory 1700 orthe learning processor 1300, or transmit the history information to anexternal device such as the AI server 200. The collected historyinformation may be used to update a learning model.

The processor 1800 may control at least some of the components of the AIdevice 1000 to run an application program stored in the memory 1700.Further, the processor 1800 may operate two or more components includedin the AI device 1000 in combination to execute the application program.

FIG. 2 illustrates an AI server according to one implementation of thepresent disclosure.

Referring to FIG. 2, the AI server 200 may refer to a device that trainsan artificial neural network using a machine learning algorithm or usesa trained artificial neural network. Here, the AI server 200 may includea plurality of servers to perform distributed processing, or be definedas a 5G network. In this case, the AI server 200 may be included as apartial configuration of the AI device 1000, so as to perform at leastpart of AI processing together.

The AI server 200 may include a communication unit 210, a memory 230, alearning processor 240, a processor 260, and the like.

The communication unit 210 may transmit and receive data with anexternal device such as the AI device 1000.

The memory 230 may include a model storage unit 231. The model storageunit 231 may store a model (or artificial neural network 231 a) that isbeing trained or has been trained by the learning processor 240.

The learning processor 240 may train the artificial neural network 231 ausing training data. A learning model may be used in a state of beinginstalled on the AI server 200 of an artificial neural network, or maybe installed on an external device such as the AI device 1000.

The learning model may be implemented as hardware, software, or acombination thereof. When some or the entire of the learning model isimplemented as software, one or more instructions constructing thelearning model may be stored in the memory 230.

The processor 260 may infer a result value for new input data using thelearning model, and generate a response or a control command based onthe inferred result value.

FIG. 3 illustrates an AI system according to one implementation of thepresent disclosure.

Referring to FIG. 3, in an AI system 1, at least one of the AI server200, a robot 100 a, an autonomous vehicle (or self-driving vehicle) 100b, an XR device 100 c, a smartphone 100 d, and a home appliance 100 e isconnected to a cloud network 10. Here, the robot 100 a, the autonomousvehicle 100 b, the XR device 100 c, the smartphone 100 d, and the homeappliance 100 e to which the AI technology is applied may be referred toas “AI devices” 100 a to 100 e.

The cloud network 10 may be a network that constitutes a part of a cloudcomputing infrastructure or exists in the cloud computinginfrastructure. Here, the cloud network 10 may be constructed using a 3Gnetwork, a 4G or LTE network, and/or a 5G network.

That is, the devices (100 a to 100 e, 200) constituting the AI system 1may be connected to each other through the cloud network 10. Inparticular, the devices 100 a to 100 e and 200 may communicate with eachother through a base station, or may directly communicate with eachother without through the base station.

The AI server 200 may include a server for performing AI processing anda server for performing calculation on big data.

The AI server 200 may be connected to at least one of the robot 100 a,the autonomous vehicle 100 b, the XR device 100 c, the smartphone 100 d,and the home appliance 100 e, which are the AI devices constituting theAI system 1, through the cloud network 10, and may help at least part ofthe AI processing of the connected AI devices 100 a to 100 e.

Here, the AI server 200 may train an artificial neural network accordingto a machine learning algorithm in place of the AI devices 100 a to 100e, and may directly store a learning model or transmit the learningmodel to the AI devices 100 a to 100 e.

At this time, the AI server 200 may receive input data from the AIdevices 100 a to 100 e, infer a result value of the received input datausing the learning model, generate a response or a control command basedon the inferred result value, and transmit the generated response orcontrol command to the AI devices 100 a to 100 e.

Alternatively, the AI devices 100 a to 100 e may directly infer a resultvalue from input data using a learning model, and generate a response ora control command based on the inferred result value.

Hereinafter, various implementations of the AI devices 100 a to 100 e towhich the above-described technologies are applied will be described.Here, the AI devices 100 a to 100 e illustrated in FIG. 3 may be aspecific example of the AI device 1000 shown in FIG. 1.

As an AI technology is applied to the robot 100 a, the robot 100 a maybe implemented as a guide robot, a transport robot, a cleaning robot, awearable robot, an entertainment robot, a pet robot, an unmanned flyingrobot, and the like.

The robot 100 a may include a robot control module for controlling anoperation, and the robot control module may refer to a software moduleor a chip which is a hardware implementation of the software module.

The robot 100 a may acquire state or status information of the robot 100a, detect (recognize) a surrounding environment and objects, generatemap data, determine a travel path (or route) and a driving plan, providea response to user interaction, or determine an operation using sensorinformation obtained from various types of sensors.

Here, the robot 100 a may use sensor information obtained by at leastone sensor from a LiDAR, a radar, and a camera in order to determine atravel path and a driving plan.

The robot 100 a may perform the operations described above using alearning model that consists of at least one artificial neural network.For example, the robot 100 a may recognize a surrounding environment andobjects using a learning model and determine an operation usingrecognized surrounding environment or object information. Here, thelearning model may have been directly trained in the robot 100 a or havebeen trained in an external device such as the AI server 200.

Here, the robot 100 a may directly generate a result using the learningmodel to perform an operation, or perform an operation by transmittingsensor information to an external device such as the AI server 200 andreceiving a result generated accordingly.

The robot 100 a may determine a travel path and a driving plan by usingat least one of object information acquired from map data, objectinformation detected from sensor information, or object informationobtained from an external device, and control the drive unit such thatthe robot 100 a travels according to the determined travel path anddriving plan.

The map data may include object identification information regardingvarious objects located in a space in which the robot 100 a travels. Forexample, the map data may include object identification informationregarding fixed objects such as a wall and a door, and movable objectssuch as a flower pot and a desk. In addition, the object identificationinformation may include a name, a type, a distance, a location (orposition), and the like.

In addition, the robot 100 a may perform an operation or travel bycontrolling the drive unit based on user's control/interaction. Here,the robot 100 a may acquire intention information of an interactionaccording to a user's motion or voice (speech), determine a responsebased on the obtained intention information, and perform an operation.

As an AI technology is employed in the autonomous vehicle 100 b, theautonomous vehicle 100 b may be implemented as a mobile robot, vehicle,or unmanned aerial vehicle.

The autonomous vehicle 100 b may include an autonomous driving controlmodule for controlling an autonomous driving function, and theautonomous driving control module may refer to a software module or achip which is a hardware implementation of the software. The autonomousdriving control module may be included in the autonomous driving vehicle100 b, or may be configured as separate hardware provided outside theautonomous vehicle 100 b to be connected thereto.

The autonomous driving vehicle 100 b may use sensor information obtainedfrom various types of sensors to obtain state information of theautonomous vehicle 100 b, detect (recognize) a surrounding environmentand objects, generate map data, determine a travel path and a drivingplan, or to determine an operation.

Like the robot 100 a, the autonomous vehicle 100 b may use sensorinformation obtained from at least one sensor among a LiDAR, a radar,and a camera to determine a travel path and a driving plan.

In particular, the autonomous vehicle 100 b may recognize an environmentor an object in an area where the field of view is blocked or an areagreater than or equal to a specific distance by receiving sensorinformation from external devices or by receiving directly recognizedinformation from the external devices.

The autonomous vehicle 100 b may perform the above-described operationsusing a learning model consisting of at least one artificial neuralnetwork. For example, the autonomous vehicle 100 b may recognize asurrounding environment and objects using a learning model, and maydetermine the flow of driving using recognized surrounding environmentinformation or object information. Here, the learning model may havebeen directly trained in the autonomous vehicle 100 b or trained in anexternal device such as the AI server 200.

Here, the autonomous vehicle 100 b may directly generate a result usingthe learning model to perform an operation, or may perform an operationby transmitting sensor information to an external device such as the AIserver 200 and receiving the result generated accordingly.

The autonomous vehicle 100 b may use at least one of map data, objectinformation detected from sensor information, and object informationobtained from an external device to determine a travel path and adriving plan, and control the drive unit such that the autonomousvehicle 100 b travels according to the determined travel path anddriving plan.

The map data may include object identification information regardingvarious objects located in a space (e.g., a road) in which theautonomous vehicle 100 b travels. For example, the map data may includeobject identification information regarding fixed objects such as astreetlight, a rock, and a building, and movable objects such as avehicle and a pedestrian. In addition, the object identificationinformation may include a name, a type, a distance, a location and thelike.

Further, the autonomous vehicle 100 b may perform an operation or travelby controlling the drive unit based on user's control/interaction. Inthis case, the autonomous vehicle 100 b may obtain intention informationof an interaction according a user's motion or voice (speech), determinea response based on the acquired intention information, and perform anoperation.

As an AI technology is applied to the XR device 100 c, the XR device 100c may be implemented as a Head-Mount Display (HMD), a Head-Up Display(HUD) provided in a vehicle, a TV, a mobile phone, a smartphone, acomputer, a wearable device, a home appliance, digital signage, avehicle, a stationary robot, and a mobile (or moving) robot.

The XR device 100 c may analyze 3D point cloud data or image dataacquired through various sensors or from an external device, generatelocation data and attribute data for 3D points, obtain informationregarding a surrounding space or real-world objects, and render an XRobject to output. For example, the XR device 100 c may output an XRobject including additional information regarding a recognized object incorresponding to the recognized object.

The XR device 100 c may perform the above-described operations using alearning model consisting of at least one artificial neural network. Forexample, the XR device 100 c may recognize a real-world object in 3Dpoint cloud data or image data using a learning model, and may provideinformation corresponding to the recognized real-world object. Here, thelearning model may have been directly trained in the XR device 100 c ormay have been trained in an external device such as the AI server 200.

Here, the XR device 100 c may directly generate a result using alearning model to perform an operation, or may perform an operation bytransmitting sensor information to an external device such as the AIserver 200 and receiving a result generated accordingly.

As an AI technology and autonomous driving technology are applied to therobot 100 a, the robot 100 a may be implemented as a guide robot, atransport robot, a cleaning robot, a wearable robot, an entertainmentrobot, a pet robot, an unmanned flying robot, etc.

The robot 100 a to which the AI technology and autonomous drivingtechnology are applied may be a robot itself having an autonomousdriving function or a robot 100 a interacting with the autonomousvehicle 100 b.

The robot 100 a having the autonomous driving function may collectivelyrefer to devices that travel by themselves according to a given trafficflow without the user's control or by determining a traffic flow totravel by themselves.

The robot 100 a having an autonomous driving function and the autonomousdriving vehicle 100 b may use a common sensing technique to determineone or more of a travel path and a driving plan. For example, the robot100 a having an autonomous driving function and the autonomous vehicle100 b may determine one or more of a travel path and a driving planusing information sensed through a LiDAR, a radar, and a camera.

The robot 100 a interacting with the autonomous vehicle 100 b may existseparately or independently from the autonomous vehicle 100 b, andperform an operation linked to an autonomous driving function at theinside or outside of the autonomous vehicle 100 b or perform anoperation associated with a user on board the autonomous vehicle 100 b.

Here, the robot 100 a interacting with the autonomous vehicle 100 b mayobtain sensor information on behalf of the autonomous vehicle 100 b andprovide it to the autonomous vehicle 100 b, or acquire sensorinformation and generate surrounding environment information or objectinformation to provide them to the autonomous vehicle 100 b, to therebycontrol or assist the autonomous driving function of the autonomousvehicle 100 b.

Alternatively, the robot 100 a interacting with the autonomous vehicle100 b may monitor a user on board the autonomous vehicle 100 b orcontrol functions of the autonomous vehicle 100 b through an interactionwith a user. For example, when it is determined that the driver is indrowsy state, the robot 100 a may activate an autonomous drivingfunction of the autonomous vehicle 100 b or assist control the driveunit of the autonomous vehicle 100 b. Here, the functions of theautonomous vehicle 100 b controlled by the robot 100 a may include notonly the autonomous driving function, but also a function provided by anavigation system or an audio system provided in the autonomous vehicle100 b.

Alternatively, the robot 100 a interacting with the autonomous drivingvehicle 100 b may provide information to the autonomous vehicle 100 b orassist a function from the outside of the autonomous vehicle 100 b. Forexample, the robot 100 a may provide the autonomous vehicle 100 b withtraffic information including signal information as in smart trafficlights, or automatically connect an automatic electric charger to acharging port through an interaction with the autonomous vehicle 100 bas in an automatic electric charger of an electric vehicle.

As an AI technology and an XR technology are applied to the robot 100 a,the robot 100 a may be implemented as a guide robot, a transport robot,a cleaning robot, a wearable robot, an entertainment robot, a pet robot,an unmanned flying robot, a drone, etc.

The robot 100 a to which the XR technology is applied may refer to arobot that is a target of control/interaction in an XR image. In thiscase, the robot 100 a is distinguished or different from the XR device100 c, and they may be interlocked with each other.

When the robot 100 a, which is a target of control/interaction in an XRimage, acquires sensor information from sensors including a camera, therobot 100 a or the XR device 100 c may generate an XR image based on thesensor information, and the XR device 100 c may output the generated XRimage. In addition, the robot 100 a may operate based on a controlsignal input through the XR device 100 c or a user's interaction.

For example, a user may check or identify an XR image corresponding tothe viewpoint of the robot 100 a remotely linked through an externaldevice such as the XR device 100 c, adjust an autonomous driving path ofthe robot 100 a through an interaction, control an operation or driving,or check information of surrounding objects.

As an AI technology and an XR technology is applied to the autonomousvehicle 100 b, the autonomous vehicle 100 b may be implemented as amobile robot, a vehicle, or an unmanned aerial vehicle, etc.

The autonomous vehicle 100 b to which the XR technology is applied mayrefer to an autonomous vehicle including means for providing an XRimage, or an autonomous driving vehicle that is a target ofcontrol/interaction in an XR image. In particular, the autonomousvehicle 100 b, which a target of control/interaction in an XR image, isdistinguished from the XR device 100 c, and they may be interlocked witheach other.

The autonomous vehicle 100 b equipped with means for providing an XRimage may obtain sensor information from sensors including a camera, andmay output an XR image generated based on the obtained sensorinformation. For example, the autonomous vehicle 100 b may include a HUDto provide an occupant on board with an XR object corresponding to areal-world object or an object in a screen by outputting an XR image.

Here, when the XR object is output to the HUD, at least a part of the XRobject may be overlaid or superimposed on a real-world object at whichthe occupant's gaze is directed. On the other hand, when the XR objectis displayed on a display provided in the autonomous vehicle 100 b, atleast a part of the XR object may be overlaid on an object in a screen.For example, the autonomous vehicle 100 b may output XR objectscorresponding to objects such as lanes, other vehicles, traffic lights,traffic signs, two-wheeled vehicles, pedestrians, and buildings.

When the autonomous vehicle 100 b, which is a target ofcontrol/interaction in an XR image, acquires sensor information fromsensors including a camera, the autonomous vehicle 100 b or the XRdevice 100 c may generate an XR image based on the sensor information,and the XR device 100 c may output the generated XR image. In addition,the autonomous vehicle 100 b may operate based on a control signal inputthrough an external device such as the XR device 100 c or a user'sinteraction.

The vehicle disclosed herein may include any of an internal combustionengine car having an engine as a power source, a hybrid vehicle havingan engine and an electric motor as power sources, an electric vehiclehaving an electric motor as a power source, and the like.

In the following description, a left side of a vehicle refers to a leftside in a driving or traveling direction of the vehicle, and a rightside of the vehicle refers to a right side in the driving direction.

FIG. 4 illustrates an outer appearance of a vehicle according to animplementation of the present disclosure.

FIG. 5 illustrates a vehicle exterior from various angles according toan implementation of the present disclosure.

FIGS. 6 and 7 illustrate a vehicle interior according to animplementation of the present disclosure.

FIGS. 8 and 9 are views referenced to describe objects according to animplementation of the present disclosure.

FIG. 10 is a block diagram illustrating a vehicle according to animplementation of the present disclosure.

As illustrated in FIGS. 4 to 10, a vehicle 100 may include wheelsturning by a driving force, and a steering input device 510 foradjusting a driving (preceding, moving) direction of the vehicle 100.

The vehicle 100 may be an autonomous vehicle.

Here, the autonomous driving is defined as controlling at least one ofacceleration, deceleration, and driving direction based on a presetalgorithm. In other words, the autonomous driving refers to that adriving control apparatus is automatically manipulated even without auser input applied to the driving control apparatus.

The vehicle 100 may be switched into an autonomous mode or a manual modebased on a user input.

For example, the vehicle 100 may be converted from the manual mode intothe autonomous mode or from the autonomous mode into the manual modebased on a user input received through a user interface apparatus 200.

The vehicle 100 may be switched into the autonomous mode or the manualmode based on driving environment information. The driving environmentinformation may be generated based on object information provided froman object detecting apparatus 300.

For example, the vehicle 100 may be switched from the manual mode intothe autonomous mode or from the autonomous module into the manual modebased on driving environment information generated in the objectdetecting apparatus 300.

For instance, the vehicle 100 may be switched from the manual mode intothe autonomous mode or from the autonomous module into the manual modebased on driving environment information received through acommunication apparatus 400.

The vehicle 100 may be switched from the manual mode into the autonomousmode or from the autonomous module into the manual mode based oninformation, data or signal provided from an external device.

When the vehicle 100 is driven in the autonomous mode, the autonomousvehicle 100 may be driven based on an operation system 700.

For example, the autonomous vehicle 100 may be driven based oninformation, data or signal generated in a driving system 710, a parkingexit system 740, and a parking system 750.

When the vehicle 100 is driven in the manual mode, the autonomousvehicle 100 may receive a user input for driving through a drivingcontrol apparatus 500. The vehicle 100 may be driven based on the userinput received through the driving control apparatus 500.

An overall length refers to a length from a front end to a rear end ofthe vehicle 100, a width refers to a width of the vehicle 100, and aheight refers to a length from a bottom of a wheel to a roof. In thefollowing description, an overall-length direction L may refer to adirection which is a criterion for measuring the overall length of thevehicle 100, a width direction W may refer to a direction that is acriterion for measuring a width of the vehicle 100, and a heightdirection H may refer to a direction that is a criterion for measuring aheight of the vehicle 100.

As illustrated in FIG. 10, the vehicle 100 may include a user interfaceapparatus 200, an object detecting apparatus 300, a communicationapparatus 400, a driving control apparatus 500, a vehicle operatingapparatus 600, an operation system 700, a navigation system 770, asensing unit 120, an interface unit 130, a memory 140, a control unit170, and a power supply unit 190.

In some implementations, the vehicle 100 may include more components inaddition to components described in this specification or may excludeone or more of the components described herein.

The user interface apparatus 200 is an apparatus for communicationbetween the vehicle 100 and a user. The user interface apparatus 200 mayreceive a user input and provide information generated in the vehicle100 to the user. The vehicle 100 may implement user interfaces (UIs) oruser experiences (UXs) through the user interface apparatus 200.

The user interface apparatus 200 may include an input unit 210, aninternal camera 220, a biometric sensing unit 230, an output unit 250,and a controller (or processor) 270.

According to some implementations, the user interface apparatus 200 mayinclude more components in addition to the components described in thisspecification or may not include some of those components describedherein.

The input unit 210 may allow the user to input information. Datacollected in the input unit 210 may be analyzed by the controller 270and processed as a user's control command.

The input unit 210 may be disposed inside the vehicle. For example, theinput unit 210 may be disposed on one area of a steering wheel, one areaof an instrument panel, one area of a seat, one area of each pillar, onearea of a door, one area of a center console, one area of a headlining,one area of a sun visor, one area of a wind shield, one area of awindow, or other suitable areas in the vehicle.

The input unit 210 may include an audio (or voice) input module 211, agesture input module 212, a touch input module 213, and a mechanicalinput module 214.

The audio input module 211 may convert a user's voice input into anelectric signal. The converted electric signal may be provided to thecontroller 270 or the control unit 170.

The audio input module 211 may include at least one microphone.

The gesture input module 212 may convert a user's gesture input into anelectric signal. The converted electric signal may be provided to thecontroller 270 or the control unit 170.

The gesture input module 212 may include at least one of an infraredsensor and an image sensor for detecting the user's gesture input.

According to some implementations, the gesture input module 212 maydetect a user's three-dimensional (3D) gesture input. To this end, thegesture input module 212 may include a light emitting diode outputting aplurality of infrared rays or a plurality of image sensors.

The gesture input module 212 may detect the user's 3D gesture input by atime of flight (TOF) method, a structured light method or a disparitymethod.

The touch input module 213 may convert the user's touch input into anelectric signal. The converted electric signal may be provided tocontroller 270 or the control unit 170.

The touch input module 213 may include a touch sensor for detecting theuser's touch input.

In some implementations, the touch input module 213 may be integratedwith a display module 251 so as to implement a touch screen. The touchscreen may provide an input interface and an output interface betweenthe vehicle 100 and the user.

The mechanical input module 214 may include at least one of a button, adome switch, a jog wheel and a jog switch. An electric signal generatedby the mechanical input module 214 may be provided to the controller 270or the control unit 170.

The mechanical input module 214 may be arranged on a steering wheel, acenter fascia, a center console, a cockpit module, a door, and/or othersuitable areas in the vehicle.

The internal camera 220 may acquire an internal image of the vehicle.The controller 270 may detect a user's state based on the internal imageof the vehicle. The controller 270 may acquire information related tothe user's gaze from the internal image of the vehicle. The controller270 may detect a user gesture from the internal image of the vehicle.

The biometric sensing unit 230 may acquire the user's biometricinformation. The biometric sensing module 230 may include a sensor fordetecting the user's biometric information and acquire fingerprintinformation and heart rate information regarding the user using thesensor. The biometric information may be used for user authentication.

The output unit 250 may generate an output related to a visual, audible,or tactile signal.

The output unit 250 may include at least one of a display module 251, anaudio output module 252, and a haptic output module 253.

The display module 251 may output graphic objects corresponding tovarious types of information.

The display module 251 may include at least one of a liquid crystaldisplay (LCD), a thin film transistor-LCD (TFT LCD), an organiclight-emitting diode (OLED), a flexible display, a three-dimensional(3D) display, and an e-ink display.

The display module 251 may be inter-layered or integrated with a touchinput module 213 to implement a touch screen.

The display module 251 may be implemented as a head up display (HUD).When the display module 251 is implemented as the HUD, the displaymodule 251 may be provided with a projecting module so as to outputinformation through an image which is projected on a windshield or awindow.

The display module 251 may include a transparent display. Thetransparent display may be attached to the windshield or the window.

The transparent display may have a predetermined degree of transparencyand output a predetermined screen thereon. The transparent display mayinclude at least one of a thin film electroluminescent (TFEL), atransparent OLED, a transparent LCD, a transmissive transparent displayand a transparent LED display. The transparent display may haveadjustable transparency.

Meanwhile, the user interface apparatus 200 may include a plurality ofdisplay modules 251 a to 251 g.

The display module 251 may be disposed on one area of a steering wheel,one area 521 a, 251 b, 251 e of an instrument panel, one area 251 d of aseat, one area 251 f of each pillar, one area 251 g of a door, one areaof a center console, one area of a headlining or one area of a sunvisor, or implemented on one area 251 c of a windshield or one area 251h of a window.

The audio output module 252 may convert an electric signal provided fromthe controller 270 or the control unit 170 into an audio signal foroutput. To this end, the audio output module 252 may include at leastone speaker.

The haptic output module 253 may generate a tactile output. For example,the haptic output module 253 may vibrate the steering wheel, a safetybelt, a seat 110FL, 110FR, 110RL, 110RR such that the user can recognizesuch output.

The controller 270 may control an overall operation of each unit of theuser interface apparatus 200.

According to some implementations, the user interface apparatus 200 mayinclude a plurality of controllers 270 or may not include the controller270.

When the controller 270 is not included in the user interface apparatus200, the user interface apparatus 200 may operate according to a controlof a controller of another apparatus within the vehicle 100 or thecontrol unit 170.

The user interface apparatus 200 may also be referred to herein as adisplay apparatus for vehicle.

The user interface apparatus 200 may operate according to the control ofthe control unit 170.

The object detecting apparatus 300 is an apparatus for detecting anobject located at outside of the vehicle 100.

The object may be a variety of objects associated with driving oroperation of the vehicle 100.

Referring to FIGS. 8 and 9, an object O may include traffic lanes OB10,another vehicle OB11, a pedestrian OB12, a two-wheeled vehicle OB13,traffic signals OB14 and OB15, light, a road, a structure, a speed hump,a terrain, an animal, and the like.

The lane OB10 may be a driving lane, a lane next to the driving lane ora lane on which another vehicle comes in an opposite direction to thevehicle 100. Each lane OB10 may include left and right lines forming thelane.

The another vehicle OB11 may be a vehicle which is moving near thevehicle 100. The another vehicle OB11 may be a vehicle located within apredetermined distance from the vehicle 100. For example, the anothervehicle OB11 may be a vehicle moving ahead of or behind the vehicle 100.

The pedestrian OB12 may be a person located near the vehicle 100. Thepedestrian OB12 may be a person located within a predetermined distancefrom the vehicle 100. For example, the pedestrian OB12 may be a personlocated on a sidewalk or roadway.

The two-wheeled vehicle OB12 may refer to a vehicle (transportationfacility) that is located near the vehicle 100 and moves using twowheels. The two-wheeled vehicle OB13 may be a vehicle that is locatedwithin a predetermined distance from the vehicle 100 and has two wheels.For example, the two-wheeled vehicle OB13 may be a motorcycle or abicycle that is located on a sidewalk or roadway.

The traffic signals may include a traffic light OB15, a traffic sign OB14, and a pattern or text drawn on a road surface.

The light may be light emitted from a lamp provided on another vehicle.The light may be light generated from a streetlamp. The light may besolar light.

The road may include a road surface, a curve, an upward slope, adownward slope, and the like.

The structure may be an object that is located near a road and fixed onthe ground. For example, the structure may include a streetlamp, aroadside tree, a building, an electric pole, a traffic light, a bridge,and the like.

The terrain may include a mountain, a hill, and the like.

In some implementations, objects may be classified into a moving objectand a fixed object. For example, the moving object may be a conceptincluding another vehicle and a pedestrian. The fixed object mayinclude, for example, a traffic signal, a road, or a structure.

The object detecting apparatus 300 may include a camera 310, a radar320, a LiDAR 330, an ultrasonic sensor 340, an infrared sensor 350, anda controller (or processor) 370.

According to some implementations, the object detecting apparatus 300may further include other components in addition to the componentsdescribed herein, or may not include some of the components describedherein.

The camera 310 may be located on an appropriate portion outside thevehicle to acquire an external image of the vehicle. The camera 310 maybe a mono camera, a stereo camera 310 a, an around view monitoring (AVM)camera 310 b or a 360-degree camera.

In some implementations, the camera 310 may be disposed adjacent to afront windshield within the vehicle to acquire a front image of thevehicle. Alternatively, the camera 310 may be disposed adjacent to afront bumper or a radiator grill.

Alternatively, the camera 310 may be disposed adjacent to a rear glasswithin the vehicle to acquire a rear image of the vehicle.Alternatively, the camera 310 may be disposed adjacent to a rear bumper,a trunk or a tail gate.

Alternatively, the camera 310 may be disposed adjacent to at least oneof side windows within the vehicle to acquire a side image of thevehicle. Alternatively, the camera 310 may be disposed adjacent to aside mirror, a fender or a door.

The camera 310 may provide an acquired image to the controller 370.

The radar 320 may include electric wave transmitting and receivingportions. The radar 320 may be implemented as a pulse radar or acontinuous wave radar according to a principle of emitting electricwaves. The radar 320 may be implemented in a frequency modulatedcontinuous wave (FMCW) manner or a frequency shift Keying (FSK) manneraccording to a signal waveform, among the continuous wave radar methods.

The radar 320 may detect an object in a time of flight (TOF) manner or aphase-shift manner through the medium of the electric wave, and detect aposition of the detected object, a distance from the detected object anda relative speed with the detected object.

The radar 320 may be disposed on an appropriate position outside thevehicle for detecting an object which is located at a front, rear, orside of the vehicle.

The LiDAR 330 may include laser transmitting and receiving portions. TheLiDAR 330 may be implemented in a time of flight (TOF) manner or aphase-shift manner.

The LiDAR 330 may be implemented as a drive type or a non-drive type.

For the drive type, the LiDAR 330 may be rotated by a motor and detectobject near the vehicle 100.

For the non-drive type, the LiDAR 330 may detect, through lightsteering, objects which are located within a predetermined range basedon the vehicle 100. The vehicle 100 may include a plurality of non-drivetype LiDARs 330.

The LiDAR 330 may detect an object in a TOP manner or a phase-shiftmanner through the medium of a laser beam, and detect a position of thedetected object, a distance from the detected object and a relativespeed with the detected object.

The LiDAR 330 may be disposed on an appropriate position outside thevehicle for detecting an object located at the front, rear, or side ofthe vehicle.

The ultrasonic sensor 340 may include ultrasonic wave transmitting andreceiving portions. The ultrasonic sensor 340 may detect an object basedon an ultrasonic wave, and detect a position of the detected object, adistance from the detected object, and a relative speed with thedetected object.

The ultrasonic sensor 340 may be disposed on an appropriate positionoutside the vehicle for detecting an object located at the front, rear,or side of the vehicle.

The infrared sensor 350 may include infrared light transmitting andreceiving portions. The infrared sensor 350 may detect an object basedon infrared light, and detect a position of the detected object, adistance from the detected object, and a relative speed with thedetected object.

The infrared sensor 350 may be disposed on an appropriate positionoutside the vehicle for detecting an object located at the front, rear,or side of the vehicle.

The controller 370 may control an overall operation of each unit of theobject detecting apparatus 300.

The controller 370 may detect an object based on an acquired image, andtrack the object. The controller 370 may execute operations, such as acalculation of a distance from the object, a calculation of a relativespeed with the object and the like, through an image processingalgorithm.

The controller 370 may detect an object based on a reflectedelectromagnetic wave, which is generated when an emitted electromagneticwave is reflected from the object, and track the object. The controller370 may execute operations, such as a calculation of a distance from theobject, a calculation of a relative speed with the object and the like,based on the reflected electromagnetic wave.

The controller 370 may detect an object based on a reflected laser beam,which is generated when an emitted laser beam is reflected from theobject, and track the object. The controller 370 may execute operations,such as a calculation of a distance from the object, a calculation of arelative speed with the object and the like, based on the reflectedlaser beam.

The controller 370 may detect an object based on a reflected ultrasonicwave, which is generated when an emitted ultrasonic wave is reflectedfrom the object, and track the object. The controller 370 may executeoperations, such as a calculation of a distance from the object, acalculation of a relative speed with the object and the like, based onthe reflected ultrasonic wave.

The controller 370 may detect an object based on reflected infraredlight, which is generated when emitted infrared light is reflected fromthe object, and track the object. The controller 370 may executeoperations, such as a calculation of a distance from the object, acalculation of a relative speed with the object and the like, based onthe reflected infrared light.

In some implementations, the object detecting apparatus 300 may includea plurality of controllers 370 or may not include the controller 370.For example, each of the camera 310, the radar 320, the LiDAR 330, theultrasonic sensor 340 and the infrared sensor 350 may include acontroller in an individual manner.

When the controller 370 is not included in the object detectingapparatus 300, the object detecting apparatus 300 may operate accordingto the control of a controller of an apparatus within the vehicle 100 orthe control unit 170.

The object detecting apparatus 300 may operate according to the controlof the control unit 170.

The communication apparatus 400 is an apparatus for performingcommunication with an external device. Here, the external device may beanother vehicle, a mobile terminal or a server. The communicationapparatus 400 may be referred to as a ‘wireless communication unit’.

The communication apparatus 400 may perform the communication byincluding at least one of a transmitting antenna, a receiving antenna,and radio frequency (RF) circuit and RF device for implementing variouscommunication protocols.

The communication apparatus 400 may include a short-range communicationunit 410, a location information unit 420, a V2X communication unit 430,an optical communication unit 440, a broadcast transceiver 450, and acontroller (or processor) 470.

In some implementations, the communication apparatus 400 may furtherinclude other components in addition to the components described herein,or may not include some of the components described herein.

The short-range communication unit 410 is a unit for facilitatingshort-range communications. Suitable technologies for implementing suchshort-range communications include BLUETOOTH™, Radio FrequencyIDentification (RFID), Infrared Data Association (IrDA), Ultra-WideBand(UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity(Wi-Fi), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), andthe like.

The short-range communication unit 410 may construct short-range areanetworks to perform short-range communication between the vehicle 100and at least one external device.

The location information unit 420 is a unit for acquiring positioninformation. For example, the location information unit 420 may includea Global Positioning System (GPS) module or a Differential GlobalPositioning System (DGPS) module.

The V2X communication unit 430 is a unit for performing wirelesscommunications with a server (Vehicle to Infra; V2I), another vehicle(Vehicle to Vehicle; V2V), or a pedestrian (Vehicle to Pedestrian; V2P).The V2X communication unit 430 may include an RF circuit implementing acommunication protocol with the infra (V2I), a communication protocolbetween the vehicles (V2V) and a communication protocol with apedestrian (V2P).

The optical communication unit 440 is a unit for performingcommunication with an external device through the medium of light. Theoptical communication unit 440 may include a light-emitting diode forconverting an electric signal into an optical signal and sending theoptical signal to the exterior, and a photodiode for converting thereceived optical signal into an electric signal.

In some implementations, the light-emitting diode may be integrated withlamps provided on the vehicle 100.

The broadcast transceiver 450 is a unit for receiving a broadcast signalfrom an external broadcast managing entity or transmitting a broadcastsignal to the broadcast managing entity via a broadcast channel. Thebroadcast channel may include a satellite channel, a terrestrialchannel, or both. The broadcast signal may include a TV broadcastsignal, a radio broadcast signal and a data broadcast signal.

The controller 470 may control an overall operation of each unit of thecommunication apparatus 400.

According to some implementations, the communication apparatus 400 mayinclude a plurality of controllers 470 or may not include the controller470.

When the controller 470 is not included in the communication apparatus400, the communication apparatus 400 may operate according to thecontrol of a controller of another device within the vehicle 100 or thecontrol unit 170.

In some implementations, the communication apparatus 400 may implement adisplay apparatus for a vehicle together with the user interfaceapparatus 200. In this instance, the display apparatus for the vehiclemay be referred to as a telematics apparatus or an Audio VideoNavigation (AVN) apparatus.

The communication apparatus 400 may operate according to the control ofthe control unit 170.

The driving control apparatus 500 is an apparatus for receiving a userinput for driving.

In a manual mode, the vehicle 100 may be operated based on a signalprovided by the driving control apparatus 500.

The driving control apparatus 500 may include a steering input device510, an acceleration input device 530, and a brake input device 570.

The steering input device 510 may receive an input regarding a driving(proceeding) direction of the vehicle 100 from the user. The steeringinput device 510 may be configured in the form of a wheel allowing asteering input in a rotating manner. In some implementations, thesteering input device 510 may be configured as a touch screen, a touchpad, or a button.

The acceleration input device 530 may receive an input for acceleratingthe vehicle 100 from the user. The brake input device 570 may receive aninput for braking the vehicle 100 from the user. Each of theacceleration input device 530 and the brake input device 570 ispreferably configured in the form of a pedal. In some implementations,the acceleration input device 530 or the brake input device 570 may beconfigured as a touch screen, a touch pad, or a button.

The driving control apparatus 500 may operate according to the controlof the control unit 170.

The vehicle operating apparatus 600 is an apparatus for electricallycontrolling operations of various devices within the vehicle 100.

The vehicle operating apparatus 600 may include a power train operatingunit 610, a chassis operating unit 620, a door/window operating unit630, a safety apparatus operating unit 640, a lamp operating unit 650,and an air-conditioner operating unit 660.

According to some implementations, the vehicle operating apparatus 600may further include other components in addition to the componentsdescribed herein, or may not include some of the components describedherein.

In some implementations, the vehicle operating apparatus 600 may includea controller. Each unit of the vehicle operating apparatus 600 mayindividually include a controller.

The power train operating unit 610 may control an operation of a powertrain device.

The power train operating unit 610 may include a power source operatingportion 611 and a gearbox operating portion 612.

The power source operating portion 611 may perform a control for a powersource of the vehicle 100.

For example, upon using a fossil fuel-based engine as the power source,the power source operating portion 611 may perform an electronic controlfor the engine. Accordingly, an output torque and the like of the enginecan be controlled. The power source operating portion 611 may adjust theengine output torque according to the control of the control unit 170.

For example, upon using an electric energy-based motor as the powersource, the power source operating portion 611 may perform a control forthe motor. The power source operating portion 611 may adjust a rotatingspeed, a torque and the like of the motor according to the control ofthe control unit 170.

The gearbox operating portion 612 may perform a control for a gearbox.

The gearbox operating portion 612 may adjust a state of the gearbox. Thegearbox operating portion 612 may change the state of the gearbox intodrive (forward) (D), reverse (R), neutral (N), or parking (P).

For example, when an engine is the power source, the gearbox operatingportion 612 may adjust a locked state of a gear in the drive (D) state.

The chassis operating unit 620 may control an operation of a chassisdevice.

The chassis operating unit 620 may include a steering operating portion621, a brake operating portion 622, and a suspension operating portion623.

The steering operating portion 621 may perform an electronic control fora steering apparatus within the vehicle 100. The steering operatingportion 621 may change a driving direction of the vehicle.

The brake operating portion 622 may perform an electronic control for abrake apparatus within the vehicle 100. For example, the brake operatingportion 622 may control an operation of brakes provided at wheels toreduce speed of the vehicle 100.

In some implementations, the brake operating portion 622 mayindividually control each of a plurality of brakes. The brake operatingportion 622 may differently control braking force applied to each of aplurality of wheels.

The suspension operating portion 623 may perform an electronic controlfor a suspension apparatus within the vehicle 100. For example, thesuspension operating portion 623 may control the suspension apparatus toreduce vibration of the vehicle 100 when a bump is present on a road.

In some implementations, the suspension operating portion 623 mayindividually control each of a plurality of suspensions.

The door/window operating unit 630 may perform an electronic control fora door apparatus or a window apparatus within the vehicle 100.

The door/window operating unit 630 may include a door operating portion631 and a window operating portion 632.

The door operating portion 631 may perform the control for the doorapparatus. The door operating portion 631 may control opening or closingof a plurality of doors of the vehicle 100. The door operating portion631 may control opening or closing of a trunk or a tail gate. The dooroperating portion 631 may control opening or closing of a sunroof.

The window operating portion 632 may perform the electronic control forthe window apparatus. The window operating portion 632 may controlopening or closing of a plurality of windows of the vehicle 100.

The safety apparatus operating unit 640 may perform an electroniccontrol for various safety apparatuses within the vehicle 100.

The safety apparatus operating unit 640 may include an airbag operatingportion 641, a seatbelt operating portion 642 and a pedestrianprotecting apparatus operating portion 643.

The airbag operating portion 641 may perform an electronic control foran airbag apparatus within the vehicle 100. For example, the airbagoperating portion 641 may control the airbag to be deployed upon adetection of a risk.

The seatbelt operating portion 642 may perform an electronic control fora seatbelt apparatus within the vehicle 100. For example, the seatbeltoperating portion 642 may control passengers to be motionlessly seatedin seats 110FL, 110FR, 110RL, and 110RR using seatbelts upon a detectionof a risk.

The pedestrian protecting apparatus operating portion 643 may perform anelectronic control for a hood lift and a pedestrian airbag. For example,the pedestrian protecting apparatus operating portion 643 may controlthe hood lift and the pedestrian airbag to be open up upon detectingpedestrian collision.

The lamp operating unit 650 may perform an electronic control forvarious lamp apparatuses within the vehicle 100.

The air-conditioner operating unit 660 may perform an electronic controlfor an air conditioner within the vehicle 100. For example, theair-conditioner operating unit 660 may control the air conditioner tosupply cold air into the vehicle when internal temperature of thevehicle is high.

The vehicle operating apparatus 600 may include a controller. Each unitof the vehicle operating apparatus 600 may individually include acontroller.

The vehicle operating apparatus 600 may operate according to the controlof the control unit 170.

The operation system 700 is a system that controls various driving modesof the vehicle 100. The operation system 700 may operate in anautonomous driving mode.

The operation system 700 may include a driving system 710, a parkingexit system 740, and a parking system 750.

In some implementations, the operation system 700 may further includeother components in addition to the components described herein, or maynot include some of the components described herein.

In some implementations, the operation system 700 may include at leastone controller. Each unit of the operation system 700 may individuallyinclude at least one controller.

In some implementations, the operation system may be implemented by thecontrol unit 170 when it is implemented in a software configuration.

In some implementations, the operation system 700 may be implemented byat least one of the user interface apparatus 200, the object detectingapparatus 300, the communication apparatus 400, the vehicle operatingapparatus 600, and the control unit 170.

The driving system 710 may perform driving of the vehicle 100.

The driving system 710 may receive navigation information from anavigation system 770, transmit a control signal to the vehicleoperating apparatus 600, and perform driving of the vehicle 100.

The driving system 710 may receive object information from the objectdetecting apparatus 300, transmit a control signal to the vehicleoperating apparatus 600, and perform driving of the vehicle 100.

The driving system 710 may receive a signal from an external devicethrough the communication apparatus 400, transmit a control signal tothe vehicle operating apparatus 600, and perform driving of the vehicle100.

The parking exit system 740 may perform an exit of the vehicle 100 froma parking lot.

The parking exit system 740 may receive navigation information from thenavigation system 770, transmit a control signal to the vehicleoperating apparatus 600, and perform the exit of the vehicle 100 fromthe parking lot.

The parking exit system 740 may receive object information from theobject detecting apparatus 300, transmit a control signal to the vehicleoperating apparatus 600 and perform the exit of the vehicle 100 from theparking lot.

The parking exit system 740 may receive a signal from an external devicethrough the communication apparatus 400, transmit a control signal tothe vehicle operating apparatus 600, and perform the exit of the vehicle100 from the parking lot.

The parking system 750 may perform parking of the vehicle 100.

The parking system 750 may receive navigation information from thenavigation system 770, and transmit a control signal to the vehicleoperating apparatus 600 to park the vehicle 100.

The parking system 750 may receive object information from the objectdetecting apparatus 300, and transmit a control signal to the vehicleoperating apparatus 600 to park the vehicle 100.

The parking system 750 may receive a signal from an external devicethrough the communication apparatus 400, and transmit a control signalto the vehicle operating apparatus 600 to park the vehicle 100.

The navigation system 770 may provide navigation information. Thenavigation information may include at least one of map information,information regarding a set destination, path information according tothe set destination, information regarding various objects on a path,lane information and current location information of the vehicle 100.

The navigation system 770 may include a memory and a controller. Thememory may store the navigation information. The controller may controlan operation of the navigation system 770.

In some implementations, the navigation system 770 may update prestoredinformation by receiving information from an external device through thecommunication apparatus 400.

In some implementations, the navigation system 770 may be classified asa sub component of the user interface apparatus 200.

The sensing unit 120 may detect a status of the vehicle. The sensingunit 120 may include a posture sensor (e.g., a yaw sensor, a rollsensor, a pitch sensor, etc.), a collision sensor, a wheel sensor, aspeed sensor, a tilt sensor, a weight-detecting sensor, a headingsensor, a gyro sensor, a position module, a vehicle forward/backwardmovement sensor, a battery sensor, a fuel sensor, a tire sensor, asteering sensor by a turn of a handle, a vehicle internal temperaturesensor, a vehicle internal humidity sensor, an ultrasonic sensor, anillumination sensor, an accelerator position sensor, a brake pedalposition sensor, and the like.

The sensing unit 120 may acquire sensing signals with respect tovehicle-related information, such as a posture, a collision, anorientation, a position (GPS information), an angle, a speed, anacceleration, a tilt, a forward/backward movement, a battery, a fuel,tires, lamps, internal temperature, internal humidity, a rotated angleof a steering wheel, external illumination, pressure applied to anaccelerator, pressure applied to a brake pedal, and the like.

The sensing unit 120 may further include an accelerator sensor, apressure sensor, an engine speed sensor, an air flow sensor (AFS), anair temperature sensor (ATS), a water temperature sensor (WTS), athrottle position sensor (TPS), a TDC sensor, a crank angle sensor(CAS), and the like.

The interface unit 130 may serve as a path allowing the vehicle 100 tointerface with various types of external devices connected thereto. Forexample, the interface unit 130 may be provided with a port connectablewith a mobile terminal, and connected to the mobile terminal through theport. In this instance, the interface unit 130 may exchange data withthe mobile terminal.

In some implementations, the interface unit 130 may serve as a path forsupplying electric energy to the connected mobile terminal. When themobile terminal is electrically connected to the interface unit 130, theinterface unit 130 supplies electric energy supplied from a power supplyunit 190 to the mobile terminal according to the control of the controlunit 170.

The memory 140 is electrically connected to the control unit 170. Thememory 140 may store basic data for units, control data for controllingoperations of units and input/output data. The memory 140 may be avariety of storage devices, such as ROM, RAM, EPROM, a flash drive, ahard drive and the like in a hardware configuration. The memory 140 maystore various data for overall operations of the vehicle 100, such asprograms for processing or controlling the control unit 170.

In some implementations, the memory 140 may be integrated with thecontrol unit 170 or implemented as a sub component of the control unit170.

The control unit 170 may control an overall operation of each unit ofthe vehicle 100. The control unit 170 may be referred to as anElectronic Control Unit (ECU).

The power supply unit 190 may supply power required for an operation ofeach component according to the control of the control unit 170.Specifically, the power supply unit 190 may receive power supplied froman internal battery of the vehicle, and the like.

At least one control unit 170 included in the vehicle 100 may beimplemented using at least one of application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and electric units performing otherfunctions.

Hereinafter, an image output device 800 provided in the vehicle 100 willbe described in detail.

The image output device 800, which is provided in the vehicle 100 may beimplemented as an independent device detachable from the vehicle 100 oras a part of the vehicle 100 which is integrally installed in thevehicle 100.

All the operation and control method of the image output device 800described in this specification may be alternatively performed by thecontrol unit 170 of the vehicle 100. That is, the operation and/orcontrol method performed by a controller 870 of the image output device800 may be performed by the control unit 170 of the vehicle 100.

Referring to FIG. 11, the image output device 800 includes acommunication unit 810, an image output unit 850, and a controller (orprocessor) 870.

The communication unit 810 is configured to perform communication withthe various components described in FIG. 10. For example, thecommunication unit 810 may receive various information provided througha controller area network (CAN). In another example, the communicationunit 810 may communicate with all devices capable of performingcommunication, such as a vehicle, a mobile terminal, a server, andanother vehicle. This may be referred to as Vehicle to everything (V2X)communication. The V2X communication may be defined as a technology ofexchanging or sharing information, such as traffic condition and thelike, while communicating with a road infrastructure and other vehiclesduring driving.

The communication unit 810 may be configured to perform communicationwith one or more devices provided in the vehicle 100. The communicationunit 810 may include a beam former and a radio frequency IC (RFIC) thatcontrols the beam former to enable 5G communication at a frequency bandof 6 GHz or higher. However, when 5G communication uses a frequency bandof 6 GHz or less, the communication unit 810 may not necessarily includethe beam former and the RFIC.

The communication unit 810 may receive information related to driving ofthe vehicle 100 from most of the devices provided in the vehicle 100.The information transmitted from the vehicle 100 to the image outputdevice 800 is referred to as ‘vehicle driving information (or vehicletravel information)’.

Vehicle driving information includes vehicle information and surroundinginformation related to the vehicle. Information related to the inside ofthe vehicle with respect to the frame of the vehicle 100 may be definedas the vehicle information, and information related to the outside ofthe vehicle may be defined as the surrounding information.

The vehicle information refers to information related to the vehicleitself. For example, the vehicle information may include a drivingspeed, a driving direction, an acceleration, an angular velocity, alocation (GPS), a weight, a number of passengers in the vehicle, abraking force of the vehicle, a maximum braking force, air pressure ofeach wheel, a centrifugal force applied to the vehicle, a driving modeof the vehicle (autonomous driving mode or manual driving mode), aparking mode of the vehicle (autonomous parting mode, automatic parkingmode, manual parking mode), whether or not a user is present in thevehicle, and information associated with the user.

The surrounding information refers to information related to anotherobject located within a predetermined range around the vehicle, andinformation related to the outside of the vehicle. The surroundinginformation of the vehicle may be a state of a road surface on which thevehicle is traveling (e.g., a frictional force), the weather, a distancefrom a preceding (succeeding) vehicle, a relative speed of a preceding(or succeeding) vehicle, a curvature of a curve when a driving lane isthe curve, information associated with an object existing in a referenceregion (predetermined region) based on the vehicle, whether or not anobject enters (or leaves) the predetermined region, whether or not theuser exists near the vehicle, information associated with the user(e.g., whether or not the user is an authenticated user), and the like.

The surrounding information may also include ambient brightness,temperature, a position of the sun, information related to a nearbysubject (a person, another vehicle, a sign, etc.), a type of a drivingroad surface, a landmark, line information, and driving laneinformation, and information required for an autonomoustravel/autonomous parking/automatic parking/manual parking mode.

In addition, the surrounding information may further include a distancefrom an object existing around the vehicle to the vehicle 100, collisionpossibility, a type of an object, a parking space for the vehicle, anobject for identifying the parking space (e.g., a parking line, astring, another vehicle, a wall, etc.), and the like.

The vehicle driving information is not limited to the example describedabove and may include all information generated from the componentsprovided in the vehicle 100.

The image output unit 850 outputs various visual information under thecontrol of the controller 870. The image output unit 850 may outputvisual information to a windshield of a vehicle or a separately providedscreen, or may output visual information through a panel. The imageoutput unit 850 may correspond to the display module 251 described withreference to FIGS. 4 to 10.

For example, the visual information output by the image output unit 850is reflected from the windshield or the screen, so that the visualinformation is displayed on the windshield or the screen. An occupant orpassenger simultaneously checks the real world located outside thevehicle 100 and a virtual object displayed on the windshield or thescreen, and augmented reality is implemented by the image output unit850.

The controller 870 performs various operations to be describedhereinafter and controls the communication unit 810 and the image outputunit 850.

The controller 870 may control one or more devices provided in thevehicle 100 through the communication unit 810.

In detail, the controller 870 may determine whether or not at least oneof a plurality of preset conditions is satisfied, based on vehicledriving information received through the communication unit 810.According to a satisfied condition, the controller 870 may control theone or more displays in different ways.

In connection with the preset conditions, the controller 870 may detectan occurrence of an event in an electrical component provided in thevehicle 100 and/or application, and determine whether the detected eventmeets a preset condition. At this time, the controller 870 may detectthe occurrence of the event from information received through thecommunication unit 810.

The application is a concept including a widget, a home launcher, andthe like, and refers to all types of programs that can be run on thevehicle 100. Accordingly, the application may be a program that performsa function of a web browser, a video playback, a messagetransmission/reception, a schedule management, or an application update.

Further, the application may include a forward collision warning (FCW),a blind spot detection (BSD), a lane departure warning (LDW), apedestrian detection (PD) A Curve Speed Warning (CSW), and aturn-by-turn navigation (TBT).

For example, the occurrence of the event may be a missed call, presenceof an application to be updated, a message arrival, start on, start off,autonomous travel on/off, pressing of an LCD awake key, an alarm, anincoming call, a missed notification, and the like.

As another example, the occurrence of the event may be a generation ofan alert set in the advanced driver assistance system (ADAS), or anexecution of a function set in the ADAS. For example, the occurrence ofthe event may be an occurrence of forward collision warning, anoccurrence of a blind spot detection, an occurrence of lane departurewarning, an occurrence of lane keeping assist warning, or an executionof autonomous emergency braking.

As another example, the occurrence of the event may also be a changefrom a forward gear to a reverse gear, an occurrence of an accelerationgreater than a predetermined value, an occurrence of a decelerationgreater than a predetermined value, a change of a power device from aninternal combustion engine to a motor, or a change from the motor to theinternal combustion engine.

In addition, even when various ECUs provided in the vehicle 100 performspecific functions, it may be determined as the occurrence of the event.

For example, when a generated event satisfies the preset condition, thecontroller 870 may control the communication unit 810 to displayinformation corresponding to the satisfied condition on one or moredisplays provided in the vehicle.

The controller 870 may transmit an autonomous driving message to atleast one of a plurality of devices provided in the vehicle 100 so as toenable autonomous driving of the vehicle 100. For example, an autonomousdriving message may be transmitted to a brake for deceleration, or anautonomous driving message may be transmitted to a steering device forchanging a driving direction.

The present disclosure enables drivers of a plurality of vehicles toshare their captured images with each other to thereby providingadditional information.

FIG. 12 is a schematic view illustrating a communication method forsharing images between vehicles, and FIG. 13 is a schematic viewillustrating image sharing between vehicles.

Referring to FIG. 12, registered vehicles transmit GPS information,captured image information, and various vehicle information to a presetor predetermined server in real time. These registered vehicles maysearch for vehicles by receiving information of other vehicles from thepreset server and stream an image or video from at least one vehicleamong the searched vehicles.

In detail, the controller 870 receives a froward image of the vehicle100 that captures an image ahead of the vehicle 100. The forward imagemay be received through the communication unit 810 and include one ormore images.

Then, the controller 870 retrieves one or more lanes on which thevehicle 100 is expected or planned to travel from the forward image.

For the sake of convenience, the one or more lanes on which the vehicle100 is expected to travel will be referred to as a “planned drivinglane”.

The planned driving lane may refer to a lane on which the vehicle 100 isexpected to travel until a time point ‘t’, which is a positive realnumber, with respect to the current time point. The ‘t’ may varyaccording to speed of the vehicle 100, characteristics of a road onwhich the vehicle 100 is traveling, and a speed limit on a road on whichthe vehicle 100 is traveling.

When the vehicle 100 is driven by autonomous driving, the planneddriving lane may refer to a lane on which the vehicle 100 is expected totravel by autonomous driving. When the vehicle 100 is driven manually,the planned driving lane may refer to a lane recommended to a driver.

In order to search for the planned driving lane, the controller 870 mayreceive a high-definition map (HD map) from a path or route providingdevice and/or a server, so as to receive vehicle driving information forspecifying (or identifying) the planned driving lane.

More specifically, the controller 870 may receive forward path or routeinformation for guiding a road ahead of the vehicle 100 in lane units(lane-by-lane).

The forward path information may provide a driving path to a destinationfor each lane drawn on the road, which may be route information inaccordance with the ADASIS protocol.

The forward path information may be provided by subdividing a path, onwhich the vehicle should travel or can travel, into lane units. Theforward path information may be information for guiding a driving pathto a destination on the lane basis. When the forward path information isdisplayed on a display mounted on the vehicle 100, a guide line forguiding a lane on which the vehicle 100 can travel may be displayed onthe map. In addition, a graphic object indicating the location of thevehicle 100 may be included on at least one lane in which the vehicle100 is located among a plurality of lanes included in the map.

For example, when the road ahead of the vehicle 100 is an 8-lane road,and the planned driving lane is a second lane, the controller 870 maysearch for the second lane in the forward image.

As another example, when the road ahead of the vehicle 100 is an 8-laneroad, and the vehicle 100 is planned to travel on a second lane from thecurrent point or location to 50 m ahead and then move to a third lane,the controller 870 may search for the second lane up to 50 m ahead andthe third lane therefrom in the forward image.

Here, searching for a lane may refer to searching for a partial areaincluding the planned driving lane in the entire area of the forwardimage. This is to allow an occupant on board the vehicle 100 tointuitively recognize the planned driving lane by displaying a carpetimage indicating the planned driving lane in a manner of overapplyingthe searched partial area.

The controller 870 outputs a carpet image or images indicating one ormore searched lanes in lane units (or lane-by-lane) through the imageoutput unit 850.

The controller 870 sets an image display area to output visualinformation based on an occupant's eye position and/or gaze.

Further, the controller 870 determines at least one of a position, size,and shape of a main carpet image based on the occupant's eye positionand/or gaze. At least one of the position, size, and shape of the maincarpet image displayed on the windshield or the screen may be changedaccording to the occupant's eye position and/or gaze. This is to providean augmented reality where the real world and a virtual image areperfectly matched.

The main carpet image that indicates the planned driving lane mayoverlap the planned driving lane and be a transparent image with apredetermined color.

The predetermined color may vary according to a reference or criterion.For example, in the case of a general road, the main carpet image may bea first color, but when snow is accumulated on the road, the main carpetimage may be a second color that is different from the first color.

Through the main carpet image, path or route information regarding alane on which the vehicle 100 driven by autonomous driving or by adriver is expected to travel may be provided to an occupant on board inlane units.

The controller 870 may provide one or more sub-carpet images, which canbe selected by an occupant, as well as the main carpet image.

The controller 870 controls the communication unit 810 to receive animage captured from another vehicle located on a path on which thevehicle 100 is expected to travel. More specifically, an image capturedfrom another vehicle may be encoded to be transmitted to the vehicle100. Accordingly, when the communication unit 810 receives the imagecaptured from the another vehicle, a separate decoding process isrequired. A decoder for decoding an encoded image may be embedded in theimage output device 800 or the vehicle 100.

The communication unit 810 shares vehicle driving information of thevehicle 100 and other vehicles via communication with a preset serverand other vehicles. The controller 870 may search for a vehicle (anothervehicle) located on a path on which the vehicle 100 is expected totravel based on a planned driving path or route of the vehicle 100 andlocation information of other vehicles.

In response to a user request, the controller 870 may receive a capturedimage from the searched vehicle in real time. The image received in realtime may be displayed together with the forward image.

For example, as illustrated in a third drawing of FIG. 13, thecontroller 870 may receive images captured from other vehicles ahead ofthe vehicle 100 traveling on a path on which the vehicle 100 is expectedto travel, so that the captured images are displayed together with aforward image 940. In the present disclosure, as images 950 and 960captured from other vehicles are displayed, road conditions may beprovided in various ways.

Further, the controller 870 controls the image output unit 850 such thatat least one of the forward image 940 and the images 950 and 960captured from the other vehicles overlaps carpet images 941, 951, and961.

The controller 870 recognizes a lane from an image captured from anothervehicle and uses recognized lane information, location information ofthe another vehicle received therefrom, and path information on whichthe vehicle 100 is expected to travel, so as to display an imageindicating a lane for the vehicle 100 to be present when the vehicle 100reaches a position where the another vehicle is currently located. Thatis, the controller 870 may control such that the image captured from theanother vehicle and the carpet image are displayed in an overlappingmanner.

As an image, captured from another vehicle present on a path on whichthe vehicle is expected to travel, and a carpet image are displayed inan overapplying manner, driving information at a longer distance can beprovided through augmented reality.

The carpet image may be overlaid or superimposed on the forward imageand the images captured from the other vehicles. Such an implementationmay be implemented in various ways.

For example, as illustrated in the third drawing of FIG. 13, thecontroller 870 may control the image output unit 850 such that theforward image 940 and the image 950 captured from another vehicle aredisplayed individually in a manner of superimposing the carpet images941 and 951 on the forward image 940 and the image 950 captured from theanother vehicle, respectively.

Here, the controller 870 may control the image output unit 850 such thata display area of the forward image 940 is larger than a display area ofthe image 950 captured from the another vehicle.

The controller 870 may display information related to the anothervehicle in addition to the image captured from the another vehicle. Forexample, as illustrated in the third drawing of FIG. 13, the controller870 may display a distance 952 between the vehicle 100 and the anothervehicle together with the image 950 captured from the another vehicle.

The controller 870 may control such that images captured from othervehicles are displayed in different sizes according to a distancebetween the vehicle 100 and the other vehicles. In other words, thecontroller 870 may reduce the displayed size of images captured from theother vehicles as the distance from the vehicle 100 increases. In someimplementations, referring to FIG. 13, the controller 870 may controlsuch that the image 950 captured from a vehicle 1.2 km away from thevehicle 100 to be displayed bigger than the image 960 captured from avehicle 5 km away from the vehicle 100.

When displaying images captured from a plurality of other vehicles, thecontroller 870 may change output or display locations of the imagesaccording to a distance between the vehicle 100 and the other vehicles.In detail, the controller 870 may arrange the images in order along onedirection (left to right, or top to bottom) in proportion to thedistance of the other vehicles from the vehicle 100.

This may allow a user to intuitively recognize the distance between hisor her vehicle and other vehicles by just checking or seeing imagescaptured from the other vehicles.

When a current lane of the vehicle 100 and a lane of another vehiclerecognized in an image captured from the another vehicle are different,the controller 870 may process the image captured from the anothervehicle based on the current lane of the vehicle 100. For example, thecontroller 870 may recognize the current lane of the vehicle 100 and thelane of the another vehicle based on objects (e.g., lanes andstructures) recognized in the images captured from the vehicle 100 andthe another vehicle. When the current lane of the vehicle 100 and thelane of the another vehicle recognized in the image captured from theanother vehicle are different, the controller 870 recognizes the currentlane of the vehicle 100 from the image captured from the anothervehicle, and processes an image such that the current lane of thevehicle 100 is located at a center of the image. Then, the controller870 controls such that the processed image is displayed.

In order to minimize the sense of incompatibility or disharmony of theprocessed image, the controller 870 may process the image such that avanishing point included in the image captured from the another vehicleis located on the current lane of the vehicle 100, rather than justcropping a portion of the image. This may allow the image captured fromthe another vehicle to be seen as if it is captured from the currentlane of the vehicle 100.

When the vehicle 100 arrives at a position where the another vehicle iscurrently located but a lane on which the vehicle 100 is expected totravel and the lane of the another vehicle recognized in the imagecaptured from the another vehicle are different, the controller 870 mayprocess the image captured from the another vehicle with respect to thelane on which the vehicle 100 is expected to travel. For example, when alane on which the vehicle 100 is expected to travel and a lane of theanother vehicle recognized in an image captured from the another vehicleare different, the controller 870 recognizes the lane on which thevehicle 100 is planned to travel and processes an image such that thelane on which the vehicle 100 is expected to travel is located at acenter of the image. Then, the controller 870 controls such that theprocessed image is displayed.

When a distance between the vehicle 100 and the another vehicle is notgreat, the forward image and an image captured from the another vehiclemay be partially the same. The controller 870 may compare the forwardimage and the image captured from the another vehicle, synthesize (mergeor combine) the forward image and the image captured from the anothervehicle when they are partially the same, then output the synthesizedimage.

During the image synthesis, the controller 870 may synthesize an imagebased on a common object included in the forward image and the imagecaptured from the another vehicle. Various objects may be included in animage captured from a moving vehicle. The objects may include, forexample, lanes, other vehicles, pedestrians, motorcycles, trafficsignals, light, roads, structures, speed bumps, landmarks, animals, andthe like. Although a specific object is stationary, a position of thespecific object changes in an image captured from a moving vehicle.

The controller 870 extracts an object having the smallest motion ormovement per unit time from the forward image and the image capturedfrom the another vehicle. Then, the controller 870 determines whetherthe objects extracted from the forward image and the image captured fromthe another vehicle are the same object. When they are the same object,the controller 870 synthesizes the two images based on the extractedobjects.

When the objects extracted from the forward image and the image capturedfrom the another vehicle are not the same object, the controller 870extracts an object having a second smallest motion per unit time fromone of the forward image and the image captured from the anothervehicle. Then, the controller 870 determines whether the object newlyextracted from the one of the forward image and the image captured fromthe another vehicle and the objects previously extracted from the oneand the other one are the same object. The controller 870 repeats theseprocesses until the same object is extracted from the two images, andthen synthesizes the two images based on the same object.

By way of further example, the controller 870 may use a map matchingfeature during the image synthesis. More specifically, the controller870 accurately matches coordinates of the vehicle 100 and the anothervehicle on map data by using GPS information of the vehicle 100 and theanother vehicle. Here, in order to increase the matching accuracy, deadreckoning may be used. Using the coordinates of the vehicle 100 and theanother vehicle on the map data, the controller 870 may calculate adistance between the vehicle 100 and the another vehicle and adifference in an image capturing angle, and the like, and synthesize twoimages based on these calculation results.

The controller 870 may control the image output unit 850 such that thesynthesized image and the carpet image are displayed in an overlappingmanner. In the present disclosure, screen information that is wider thanthe angle of view of the camera may be provided to the driver.

According to the present disclosure, not only a path on which thevehicle 100 is expected to travel but also a path on which the anothervehicle is expected to travel may be guided through carpet images. Indetail, the controller 870 receives a path on which the another vehicleis expected to travel from the another vehicle, and controls the imageoutput unit 850 such that a first carpet image indicating the path onwhich the vehicle 100 is expected to travel and a second carpet imageindicating the path on which the another vehicle is planned to travelare overlapped with or superimposed on the synthesized image.

Here, the first carpet image and the second carpet image may bedisplayed in different shapes or manners. For example, the first andsecond carpet images may be displayed in different colors or differentpatterns. This may enable a driver to intuitively distinguish a route ofhis or her own vehicle from a route of another vehicle.

In some implementations, the first and second carpet images may bedisplayed in different thicknesses. The controller 870 may displaycarpet images such that the first carpet image has the constant oridentical thickness and the second carpet image has the thickness ininverse proportion to a distance between the vehicle 100 and the anothervehicle, allowing the driver to intuitively recognize the distancebetween his or her vehicle and the another vehicle.

Further, when a planned driving path of the vehicle 100 and a planneddriving path of the another vehicle are the same, the controller 870 maycontrol the image output unit 850 such that a third carpet image havinga different shape from the first and second carpet images issuperimposed on the synthesized image.

When the driving path of at least one of the vehicle 100 and the anothervehicle is changed while the third carpet image is being displayed, thecontroller 870 stops the display of the third carpet image and controlsthe image output unit 850 such that the first and second carpet imagesare superimposed on the synthesized image.

In the present disclosure, confusion of a driver may be avoided orreduced by minimizing the display of carpet images when a route of hisor her own vehicle and a route of the another vehicle are the same.

In the present disclosure, a first carpet image indicating a path onwhich the vehicle 100 is expected to travel is superimposed on theforward image, and a second carpet image indicating a path on which theanother vehicle is expected to travel is overlaid on an image capturedfrom the another vehicle. This may allow a driver to predict a path ofanother vehicle in advance and select an appropriate lane to drive.

Further, the controller 870 may control the image output unit 850 suchthat an image captured from the another vehicle overlaps the firstcarpet image and the second carpet image. This may allow the driver tocheck both a driving path of his or her own vehicle and a driving pathof the another vehicle in the image captured from the another vehicle.

As described above, in the present disclosure, a driving path is guidedor provided to a driver by displaying a forward image captured from hisor her own vehicle and an image captured from another vehicle together,and by superimposing carpet images indicating a planned driving path onthe captured images.

Hereinafter, an example of performing an image sharing request toanother vehicle through the image display device according to thepresent disclosure will be described.

The controller 870 may display a list of other vehicles present on adriving path of the vehicle 100 in a partial display area of the imageoutput unit 850.

For example, as illustrated in a first drawing of FIG. 13, thecontroller 870 may control the image output unit 850 such that a mapimage 900 a is displayed thereon, and a graphic object 910 indicating alocation of the vehicle 100 and graphic objects 920 a and 920 bindicating locations of other vehicles are displayed on the map image900 a. When a user input is applied to the graphic object 920 a or thegraphic object 920 b indicating the locations of the other vehicles, thecontroller 870 transmits an image sharing request to a vehiclecorresponding to the graphic object to which the user input is applied.

Here, shapes of the graphic objects indicating the locations of theother vehicles may vary according to current communication state orcondition of the other vehicles. Information related to the othervehicles may be displayed together with the graphic objects. Morespecifically, the information related to the other vehicles may includecommunication condition, a distance between the vehicle 100 and othervehicles, whether or not an image captured from other vehicles iscombined with another image.

For example, referring to FIG. 13, together with the graphic objects 920a and 920 b indicating the locations of other vehicles, the controller870 may display types of wireless communication standards (5G or 4G)available in other vehicles and a distance from other vehicles. Indetail, when another vehicle that is 4 km away from the vehicle 100 isnot available for 5G communication and is only available for 4Gcommunication, the controller 870 may output “4G, 4 km ahead” togetherwith the graphic object 920 b indicating the location of the anothervehicle.

Further, when communication signal strength of another vehicle fallsbelow a preset value, the controller 870 may display a separate graphicobject indicating a delay in the image.

In the present disclosure, as communication condition of another vehicleis informed to a driver, allowing the driver to determine whether or notimage sharing with the another vehicle is smooth.

By way of further example, the controller 870 may display informationrelated to other vehicles together with a list of other vehicles presenton a driving path of the vehicle 100. In response to a user input to thelist, the controller 870 transmits an image sharing request to at leastone of the vehicles included in the list. Here, the controller 870 mayarrange the list such that vehicles using a higher standard than thecommunication standard of an own vehicle (the vehicle 100) are given ahigher priority on the list.

In the present disclosure, as information related to other vehicleslocated on a path on which a driver's or user's vehicle is expected totravel is displayed in an intuitive manner, allowing the driver toeasily select a vehicle (another vehicle) to share an image.

When the user requests for an image sharing with another vehicle that isin poor communication condition, the controller 870 may enlarge an imagecaptured from the another vehicle to display. In detail, while imagereception from the another vehicle is stopped, the controller 870 maygradually enlarge the last displayed image. This may provide an effectthat the another vehicle becomes closer to the own vehicle in a statethat the another vehicle is stationary when the image reception is notavailable due to the poor communication condition of the anothervehicle.

In the present disclosure, specific rewards or compensation is providedto another vehicle when requesting image sharing to the another vehicle.More specifically, referring to a second drawing of FIG. 13, thecontroller 870 controls the image output unit 850 to output a message930 indicating the use of points when the driver requests for sharing animage captured from the another vehicle.

In addition, the controller 850 may transmit (reward) points informationcorresponding to points offered to the another vehicle and receive animage captured from the another vehicle. The another vehicle maytransmit its captured image only when receiving preset or predeterminedpoints information.

Hereinafter, the entire process in which the controller searches fornearby vehicles, receives an image sharing request from the user, anddisplays an image captured from another vehicle will be described indetail with reference to the accompanying drawings.

FIG. 14 is a flowchart of an exemplary method of image sharing betweenvehicles, and FIG. 15 is a flowchart of an exemplary method of usingpoints for image sharing between vehicles.

First, referring to FIG. 14, the controller 870 searches for vehiclesonly within a specific distance from the own vehicle. This is because animage captured from a vehicle located too far away from the own vehiclemay not be useful or helpful to the driver.

When route navigation is used in the own vehicle, the controller 870only filters vehicles present on the route of the own vehicle. When theroute navigation is not used in the own vehicle, the controller 870filters vehicles present on a road on which the own vehicle can travel.

When the number of searched vehicles exceeds a predetermined number, thecontroller 870 may filter the searched vehicles according to apredetermined reference. Here, the controller 870 may filter thesearched vehicles such that a distance between the own vehicle and thefiltered vehicles gradually increases. Accordingly, other vehicleslocated at various distances from the own vehicle are provided ascandidates for streaming.

Then, the controller 870 determines whether an image captured fromanother vehicle is currently being streamed. When an image captured fromthe another vehicle is already being streamed, the controller 870 maydisplay a graphic object indicating ‘end streaming’ together with astreaming image. When a user input is applied to the graphic object, thecontroller 870 may end streaming.

On the other hand, when an image captured from another vehicle is notbeing streamed, the controller 870 outputs a filtered list of othervehicles and displays information related to the other vehicles includedin the list.

Finally, when a vehicle for receiving a streaming image is selected bythe driver, the controller 870 transmits points information to theselected vehicle. The controller 870 starts streaming an image uponreceiving an image captured from the selected vehicle.

Hereinafter, an example of transmitting and receiving points informationwhen sharing an image will be described with reference to theaccompanying drawings.

Referring to FIG. 15, when the controller 870 receives a streamingrequest, the controller 870 determines whether a certain amount ofpoints are registered to the own vehicle or the driver. Pointsinformation of the own vehicle or the driver may be received from apredetermined server.

When the own vehicle or the driver has enough points available for use,the controller 870 uses points registered to the own vehicle or thedriver, and transmits points information corresponding to the pointspaid to the another vehicle or the predetermined server.

When the points information is transmitted to the predetermined server,the predetermined server transmits a message informing that the pointsinformation has been transferred to the another vehicle.

When the another vehicle receives the points information from the ownvehicle or the predetermined server, an image captured therefrom istransmitted to the own vehicle.

Points paid to the another vehicle may vary according to the size ofdata streamed from the another vehicle to the own vehicle. As a streamtime increases, the amount of points paid by the own vehicle mayincrease.

The controller 870 may calculate the amount of data periodicallyreceived while streaming an image captured from the another vehicle, andpay points corresponding to the calculated data to the another vehicle.

When points are not paid from the own vehicle, the another vehicle maystop the image transmission.

The points may be separately purchased by the driver, or points receivedfrom other vehicles may be used. More specifically, the driver mayreceive points from another vehicle by providing its forward image tothe another vehicle. Points paid in this way may be used to streamimages taken by other vehicles.

Hereinafter, a method of transmitting and receiving data betweenvehicles using the communication unit will be described in more detail.

FIG. 16 is a conceptual diagram illustrating data transmission andreception between vehicles.

Referring to FIG. 16, a plurality of vehicles periodically transmitsGPS, heading, and speed information to a vehicle information server.

The controller 870 receives GPS information of other vehicles from thevehicle information server and searches for other vehicles locatedwithin a specific distance from the own vehicle.

Then, when another vehicle for streaming an image is selected by thedriver, the controller 870 transmits information of the selected vehicleand a streaming request to the vehicle information server.

When the streaming request is received by the vehicle informationserver, the streaming request and an address of a streaming server aretransmitted to the selected vehicle.

When the selected vehicle receives the streaming request, its capturedimage is transferred to the streaming server.

The vehicle information server transmits the address of the streamingserver to the own vehicle. In addition, the vehicle information servertransmits driving information of the selected vehicle to the ownvehicle.

The controller 870 sends a streaming request to the streaming serverusing the address of the streaming server. The streaming servertransmits an image taken by the selected vehicle to the own vehicle, andthe controller 870 uses the image received from the streaming server andthe driving information of the selected vehicle received from thevehicle information server to generate and display an augmented realityimage.

At this time, the controller 870 calculates calibration of a camera ofthe selected vehicle by using a calibration result calculated from acamera image of the own vehicle. Hereinafter, a method of calibrating acamera of another vehicle and a camera of a driver's vehicle will bedescribed with reference to the accompanying drawings.

FIG. 17 is a flowchart of an exemplary method of calibrating a vehiclecamera.

Referring to FIG. 17, when calibrating a camera of a preceding vehicle(a vehicle ahead) (S401), the controller 870 determines whethercalibration on a camera of an own vehicle is completed (S402).

When the calibration of the own vehicle has not been completed, thecontroller 870 performs calibration on the own vehicle (S403). Morespecifically, the controller 870 receives a forward image from thecamera (S404).

Then, the controller 870 detects a Vanishing Line (V), a Bonnet Line(B), and a Center Line (C) from the forward image (S406), and storescalibration parameters of the own vehicle (S407).

Finally, the controller 870 calculates a project matrix of the ownvehicle using the calibration parameters of the own vehicle (S408), andthen finishes the calibration of the own vehicle (S409).

Thereafter, the controller 870 receives an image captured from thecamera of the preceding vehicle (S410), and detects a Vanishing Line(V), a Bonnet Line (B) and a Center Line (C) from the received image(S411).

The controller 870 determines whether the Vs, Bs, and Cs of the ownvehicle and the preceding vehicle are the same (S413), and adjustscalibration parameters of the own vehicle until the Vs, Bs, and Cs ofthe own vehicle and the preceding vehicle are identical (S412). Then,the controller 870 recalculates the project matrix of the own vehicleusing the adjusted calibration parameters of the own vehicle (S414), andperforms calibration of the camera of the preceding vehicle based onthis (S415).

Images received from a plurality of vehicles may be displayed togetheron the image display device. Hereinafter, an example of displayingimages received from a plurality of vehicles together with an image ofan own vehicle will be described in detail.

FIGS. 18 and 19 are schematic views illustrating an example ofdisplaying images received from a plurality of vehicles together.

Referring to FIG. 18, the image output unit 850 may display images 1020and 1030 received from vehicles located at different positions from aforward image 1010 of an own vehicle. Carpet images 1011, 1021, and 1031indicating a planned driving path of the own vehicle may be superimposedon the images, respectively.

In addition, graphic objects 1022 and 1032 indicating a distance betweenthe own vehicle and other vehicles may be displayed on the image outputunit 850. Also, a progress bar 1040 indicating a relative distancebetween the other vehicles may be displayed on the image output unit850. When the distance between the other vehicles reaches 0, the displayof one of the images received from the other vehicles may be stopped.

In some implementations, as illustrated in FIG. 19, the controller 870may arrange and display a plurality of images in a vertical direction inconsideration of a driver's gaze.

Hereinafter, an example of synthesizing a plurality of images receivedfrom other vehicles in the image display device according to the presentdisclosure will be described in detail.

FIG. 20 is a flowchart of synthesizing a plurality of images receivedfrom other vehicles.

When a distance between two other vehicles (first and second vehicles)becomes closer within a predetermined distance, the controller 870determines that the two vehicles have exceeded a threshold value andcombines images captured from the two vehicles.

More specifically, referring to FIG. 20, the controller 870 determineswhether the first vehicle has exceeded a threshold value (S501).

When the first vehicle has not exceeded the threshold value, thecontroller 870 determines whether images received from the first vehicleand the second vehicle have been previously merged or synthesized(S510). When they have been previously synthesized, the controller 870splits the two images to display (S512). On the other hand, when theyhave not been previously synthesized, the controller 870 finishes imagesynthesis.

When the first vehicle has exceeded the threshold value, the controller870 determines whether there is a common area between the two images(S502). If there is no common area, the controller 870 only displays theimage received from the first vehicle (S503).

When there is a common area, the controller 870 starts merging the twoimages (S504). Here, the controller 870 calculates a common area of thetwo images (S505), and combines the common area of the image receivedfrom the second vehicle to perfectly or precisely overlap the imagereceived from the first vehicle (S506). Then, the controller 870displays the image received from the second vehicle upright to be closerto a right angle than the image received from the first vehicle (S507).The controller 870 transmits a texture image and coordinates of thesynthesized image so as to be displayed on the image output unit 850.

When there is a plurality of synthesizable images in the plurality ofimages received from the two other vehicles, the controller 870 maygenerate a plurality of synthesized images from one of the plurality ofimages and display a list of the plurality of synthesized images. Then,the controller 870 displays an image selected by a user among the imagesincluded in the list.

In the present disclosure, images of two nearby vehicles are synthesizedto thereby provide a wider field of view to the driver.

According to the present disclosure, when an own vehicle reaches thedestination, image or video streaming may be ended without a separateuser request. In detail, when the own vehicle is located within apredetermined distance from the destination, the controller 870 mayterminate the output of the received image. Thus, unnecessary data usagemay be minimized.

According to the present disclosure, when another vehicle arrives at adestination of the own vehicle, a destination image captured from theanother vehicle reaching the destination may be displayed until the ownvehicle reaches the destination. In detail, when at least one vehicleamong other vehicles is located within a predetermined distance from thedestination, the controller 870 displays the destination captured fromthe at least one vehicle until the own vehicle is located within thepredetermined distance from the destination.

Images after the destination captured from other vehicles are not usefulto a driver. In the present disclosure, an image captured when anothervehicle reaches a destination of an own vehicle is continuouslydisplayed, which may be useful for the driver to reach the destination.

In some implementations, when a specific vehicle reaches a destinationof the own vehicle while streaming an image captured by the specificvehicle, the controller 870 may perform streaming to another vehiclethat has not reached the destination.

In some implementations, when an image received from another vehiclesatisfies a preset condition, the controller 870 may output a warningmessage to the image output unit 850. In detail, the controller 870 mayrecognize a situation or event in an image received from the anothervehicle. When an accident event is detected, the controller 870 maydisplay a warning message indicating an ‘accident’. This may give moretime for the driver to respond to the accident.

According to the present disclosure, a specific image may be enlarged byuser's selection while displaying images taken from a plurality ofvehicles.

FIG. 21 is a schematic view illustrating an example in which a specificimage is displayed in a larger size by user selection.

Referring to FIG. 21, a forward image 1310 captured from an own vehicleand images 1320 and 1330 received from other vehicles may be displayedon the image output unit 850. When a user input is applied to one of theimages, the controller 870 may enlarge an image to which the user inputis applied so that the image is displayed in the largest size.

As described above, according to the present disclosure, an occupant onboard a vehicle driven by autonomous driving or by a driver may receiveroute information in lane units through a carpet image.

In addition, according to the present disclosure, an occupant on board avehicle may be provided with more various driving information throughimage information collected from other vehicles ahead of the vehicle.

The present disclosure can be implemented as computer-readable codes(applications or software) in a program-recorded medium. The method ofcontrolling the autonomous vehicle can be realized by a code stored in amemory or the like.

The computer-readable medium may include all types of recording deviceseach storing data readable by a computer system. Examples of suchcomputer-readable media may include hard disk drive (HDD), solid statedisk (SSD), silicon disk drive (SDD), ROM, RAM, CD-ROM, magnetic tape,floppy disk, optical data storage element and the like. Also, thecomputer-readable medium may also be implemented as a format of carrierwave (e.g., transmission via an Internet). The computer may include theprocessor or the controller. Therefore, it should also be understoodthat the above-described implementations are not limited by any of thedetails of the foregoing description, unless otherwise specified, butrather should be construed broadly within its scope as defined in theappended claims. Therefore, all changes and modifications that fallwithin the metes and bounds of the claims, or equivalents of such metesand bounds are therefore intended to be embraced by the appended claims.

1. An image output device provided in a vehicle to enable augmentedreality, the device comprising: a controller configured to receive, inreal time, a forward image capturing an image in front of the vehicle,detect and save a calibration parameter for calibrating the capturedforward image to calibrate the captured forward image using thecalibration parameter, search for one or more lanes on which the vehicleis expected to travel in the forward image, generate image informationincluding a carpet image or images indicating the searched one or morelanes in lane units, and transmit the image information to an imageoutput unit outputting visual information, so that the image informationis output to the image output device, wherein the controller receives,in real time, an image captured from another vehicle present on a routeon which the vehicle is expected to travel, performs calibration on theimage captured from the another vehicle using the calibration parameter,and generates image information in which the carpet image indicating theroute on which the vehicle is expected to travel is superimposed on theimage captured from the another vehicle.
 2. The device of claim 1,wherein the controller generates image information in which the carpetimage is superimposed on the forward image and the image captured fromthe another vehicle, and outputs the forward image on which the carpetimage is superimposed together with the image captured from the anothervehicle on which the carpet image is superimposed to the image outputunit.
 3. The device of claim 2, wherein the controller, when the forwardimage and the image captured from the another vehicle are at leastpartially the same, combines the forward image and the image capturedfrom the another vehicle and generates image information in which thecarpet image is superimposed on the combined image.
 4. The device ofclaim 3, wherein the controller receives a route on which the anothervehicle is expected to travel from the another vehicle and generatesimage information in which a first carpet image indicating the route onwhich the vehicle is expected to travel and a second carpet imageindicating the route on which the another vehicle is expected to travelare superimposed on the combined image.
 5. The device of claim 3,wherein the controller generates image information in which a thirdcarpet image having a different shape from the first and second carpetimages is superimposed on the combined image when the vehicle and theanother vehicle are expected to travel on the same route.
 6. The deviceof claim 5, wherein the controller, when the route of at least one ofthe vehicle and the another vehicle is changed while the third carpetimage is displayed on the image output unit, stops the generation of theimage information including the third carpet image, and creates an imagein which the first and second carpet images are superimposed on thecombined image.
 7. The device of claim 1, wherein the controllerreceives a route on which the another vehicle is expected to travel fromthe another vehicle and generates image information in which a firstcarpet image indicating the route on which the vehicle is expected totravel is superimposed on the forward image, and a second carpet imageindicating the route on which the another vehicle is expected to travelis superimposed on the image captured from the another vehicle.
 8. Thedevice of claim 7, wherein the controller generates image information inwhich the first carpet image and the second carpet image aresuperimposed on the image captured from the another vehicle.
 9. Thedevice of claim 8, wherein the controller generates image information inwhich the first carpet image and the second carpet image aresuperimposed on the forward image.
 10. The device of claim 1, whereinthe controller receives information regarding the another vehicle fromthe another vehicle, generates image information including theinformation regarding the another vehicle and the image captured fromthe another vehicle, and transmits the generated image information tothe image output unit.
 11. The device of claim 1, wherein the controllerreceives a request for image sharing of the another vehicle from adriver of the vehicle, generates a message for indicating the use ofpoints upon receiving the image sharing request, and transmits thegenerated message to the image output unit.
 12. The device of claim 1,wherein the controller transmits information about points paid to theanother vehicle, so as to receive the image captured from the anothervehicle.
 13. The device of claim 1, wherein the controller receives, inreal time, images captured from a plurality of vehicles present on theroute on which the vehicle is expected to travel, transmits the forwardimage and the images captured from the plurality of vehicles to theimage output unit, and generates image information in which the carpetimage is superimposed on at least one of the forward image and each ofthe images captured from the plurality of vehicles.
 14. The vehicle ofclaim 13, wherein the controller generates image information such thatthe images captured from the plurality of vehicles are displayed in asmaller size than the forward image.
 15. An image output device providedin a vehicle to enable augmented reality, the device comprising: animage output unit configured to output visual information for enablingthe augmented reality; a communication unit configured to communicatewith another vehicle and a server, and receive, in real time, a forwardimage capturing an image in front of the vehicle; and a controllerconfigured to detect and save a calibration parameter for calibratingthe captured forward image to calibrate the captured forward image usingthe calibration parameter, and to control the image output unit tosearch for one or more lanes on which the vehicle is expected to travelin the forward image, and output a carpet image or images indicating thesearched one or more lanes in lane units, wherein the controllercontrols the communication unit such that an image captured from anothervehicle present on a route on which the vehicle is expected to travel isreceived in real time, wherein the controller performs calibration onthe image captured from the another vehicle using the calibrationparameter, and wherein the controller controls the image output unitsuch that the carpet image indicating the route on which the vehicle isexpected to travel is superimposed on the image captured from theanother vehicle is generated.
 16. A method for controlling an imageoutput device provided in a vehicle to enable augmented reality, themethod comprising: receiving, in real time, a forward image capturing animage in front of the vehicle; detecting and saving a calibrationparameter for calibrating the captured forward image to calibrate thecaptured forward image using the calibration parameter; searching forone or more lanes on which the vehicle is expected to travel in theforward image; receiving, in real time, an image captured from anothervehicle present on a route on which the vehicle is expected to travel;calibrating the image captured from the another vehicle using thecalibration parameter; generating image information including a carpetimage or images indicating the searched one or more lanes in lane unitsusing at least one of the forward image and the image captured from theanother vehicle; and transmitting the image information to an imageoutput unit, wherein the image information includes an image in whichthe carpet image indicating the route on which the vehicle is expectedto travel is superimposed on the image captured from the anothervehicle.
 17. The method of claim 16, wherein the image informationincludes an image in which the carpet image is superimposed on theforward image and the image captured from the another vehicle, andwherein the method further comprises outputting the forward image onwhich the carpet image is superimposed together with the image capturedfrom the another vehicle on which the carpet image is superimposed tothe image output unit.
 18. The method of claim 17, further comprising:combining the forward image and the image captured from the anothervehicle when the forward image and the image captured from the anothervehicle are at least partially the same; and transmitting the combinedimage to the image output unit.
 19. The method of claim 18, furthercomprising: receiving a route on which the another vehicle is expectedto travel from the another vehicle; generating image information inwhich a first carpet image indicating the route on which the vehicle isexpected to travel and a second carpet image indicating the route onwhich the another vehicle is expected to travel are superimposed on thecombined image; and transmitting the generated image information to theimage output unit.
 20. The method of claim 18, further comprising, whenthe vehicle and the another vehicle are expected to travel on the sameroute: generating image information in which a third carpet image havinga different shape from the first and second carpet images issuperimposed on the combined image; and transmitting the generated imageinformation to the image output unit.